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EXECUTIVE SUMMARY 


This handbook explains the basic Delphi methodology and discusses modified 
Delphi techniques to identify long-term national telecommunications needs and priorities. 
Also addressed are the selection of communications experts to participate in such a 
study, the construction of questionnaires on potential communications developments, 
and requisite technology. The handbook does not recommend any one of the many 
modified Delphi approaches as preferrable. No two modified Delphi studies have been 
the same, which reflects the flexibility and adaptability of this technique. Each study 
must be specifically tailored to the particular case. 

The Delphi method, originally developed and applied at the Rand Corporation, was 
defined as the iterated interrogation of a panel of experts to obtain a systematic con- 
sensus of informed opinion. 

The procedure consists of seeking a consensus of opinion among experts about a 
particular subject and attendant conditions that may prevail in the future. The main 
features of the Delphi method are : 

1. No member of the panel of experts knows what another member of the 
panel says about a particular question (anonymity of voting). However, 
this does not mean that the identity of the other members is not known 
or at least their backgrounds. 

2. Questions are asked in two or more iterations (four is often preferred). 

They may be the same on each round and thus force a reexamination of 
views from the members of the panel; or they may consist of new questions 
raised by earlier responses. 

3. At each iteration, additional information is given to the participants, in 
the form of statistics on the earlier responses of the group as a whole, 
or the answers and comments themselves. This feedback mechanism 
might also include additional data obtained outside the group. 
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The Delphi method is an important attempt to bring rigor and logic to forecasting 
by obtaining the consensus of a number of experts in a climate that eliminates the 
influence of personalities. Where hard fact or collected data are lacking, the best 
source of information is the expert. Delphi is a proven method for systematic gathering 
of estimates and opinions. 

Delphi techniques have been applied in a number of long-range forecasting studies, 
projecting future events or developments, and their impact. These studies cover a wide 
range of social, economic, political, scientific, and technological subjects. Several 
pertinent Delphi studies were included as illustrative examples in this handbook. 

While none of these Delphi examples employed a computerized approach the use of 
computers is discussed in detail. A computerized approach to the Delphi study could 
significantly shorten its duration. However, caution must be exercised to prevent the 
overlapping of answers in response to one iteration of questions with another. This is 
essential due to the sequential iterative approach inherent in the Delphi method. 

While properly managed Delphi exercises have been considered to be highly 
successful and, whereas, the Delphi process appears to be one of the most promising 
approaches developed for social, political, and technical forecasting, it is certainly 
not advanced as a panacea. 

There are several fundamental limitations to be aware of in performing a Delphi 
study. The administrators of the study may be knowingly or unknowingly biased. This 
bias can be reflected in the selection of experts, the composition of the questionnaires 
for the experts to respond to, and in the evaluation and presentation of a round of 
answers to the questions. Questions can be designed to cope with bias on the part of 
experts but minimizing bias on the part of study administrators is more difficult. 

The experts to serve on the panel must be selected with the utmost care, not 
only to assure informed and knowledgeable responses but, to minimize the possibility 
of some experts "going along with the crowd. " This could prevent the study from 
reaching a valid consensus (or possibly equally valid, a lack of consensus). 
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A further consideration to note is that the assumption in Delphi of linear indepen- 
dence between forecasts or one set of answers and another may not be valid. If the 
likelihood of a certain event occurring changes, given the occurrence of another event, 
then, the events are not independent and the conditional probabilities associating these 
events should be considered in the statistical analysis of answers to questions regarding 
these events. Cross-impact analysis, discussed in some detail in this handbook provides 
a method for addressing the latter situation. 

To protect against the foregoing deficiencies, the final results of the study to 
determine long-range national telecommunications needs and priorities might well be 
reviewed by higher level management personnel outside of as well as within NASA. 

The statement of work guiding the development of this handbook is reproduced in 
the following paragraphs. 

The contractor shall perform a study to develop a plan for identification of long- 
term national communications needs and priorities by use of a modified Delphi 
technique. The study will include the following tasks: 

1. The plan will describe the basic Delphi technique. References to a 
relevant recent bibliography on the Delphi technique will be collected 
and identified along with the mathematical basis for the Delphi tech- 
nique, Modifications to adapt the technique to the discipline of com- 
munications technology will be specifically addressed. 

2. The plan will present the objectives that will provide for informing a 
panel of communications experts in the actual implementation of the 
Delphi study technique. The methodology presented will be directed 
toward the simplification of the initiation, implementation, and feed- 
back controls for the NASA Plan Leader. Sources for identifying 
potential communications developments and requisite technology will 
be suggested by the contractor as a result of a search of literature 

and other information media. Sources for social, political, and economic 
indicators will also be suggested. Examples of charts and statistical 
extrapolations will be included. 
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3. The plan will include modification of the original Delphi technique 
and its later revisions, such as: 

a. Suggestions on personal probability assessments in relation to 
future communications technology will be presented. Techniques 
for scaling of verbal phases associated with numerical probability 
will be included. 

b. Provisions for protections against a spurious concensus of multi- 
disciplinary judgments will be provided as caveats to the Planning 
Leader. Feedback - both critical and favorable - on past Delphi 
studies will be noted to avoid past errors. 

c. A discussion on cross-impact analysis will be included to assure 
recognition of interdependence among anticipated telecommunications 
developments. Uses of Monte Carlo simulation in Delphi sensitivity 
analysis will also be addressed. Risk Analysis Technology will be 
integrated into the overall plan. 

4. The plan will include typical condensed analyses and evaluations of 
Delphi applications to offer insight into problem areas that may arise. 

5. The plan will include the detailed procedures to be used for performing 
the actual Delphi planning activity. Included will be procedures for 
selecting the experts to be used and their desired characteristics. 

6. The plan summary and recommendations for its implementation will 
be included. 
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SECTION 1 - INTRODUCTION 


1.1 PURPOSE OF STUDY PLAN 

The purpose of this study plan is to provide NASA with a handbook to facilitate 
implementation of a program to identify long-term telecommunications needs and pri- 
orities in the national interests by use of a modified Delphi technique. National tele- 
communication needs are not confined to the 50 conterminous states but also include 
elements interactive with other nations. 

1.2 SCOPE OF STUDY PLAN 

This study plan focuses . on the use of the Delphi method with selected modifica- 
tions or developments in the prediction of long-term (1985-1995) national telecommuni- 
cation needs and priorities in general and communication satellites in particular. The 
Delphi technique was evolved at the RAND Corporation by Dr. Olaf Helmer in collab- 
oration with Dr. Norman C. Dalkey. 1 Originally it was developed and applied as a 

method for deriving a consensus of expert opinion. The first application of the method 

2 

to a major long-range forecasting study was by Dr. Helmer and Theodore J. Gordon 
of Douglas Aircraft under the auspices of RAND Corporation. The Delphi method was 
defined at RAND as a set of techniques for soliciting and collating the opinion of ex- 
perts in order to arrive at the most reliable consensus. The method's most distinc- 
tive features are anonymity, statistical summaries of information provided by the con- 
trol group, regulated feedback, and an iterative process that permits and encourages 
the reassessment of initial judgments. Delphi techniques, both basic and modified, 
are presented in this study plan with added emphasis on selection of communications 
experts and construction of questionnaires on potential communications developments 
and requisite technology. A number of Delphi studies are subsequently presented 
which illustrate selection of experts and types of questions, i.e. , in structure and in 
communications subject content. 

Key terms used in defining the scope of this study plan are elaborated in the fol- 
lowing sentences. 
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By '‘prediction" we mean the declaration beforehand of events, implying inference 
from facts or accepted laws of nature. A prediction is a statement made about the 
future or an as yet unobserved event: a reasonably definite forecast about the future. 

"Long-term" implies long-range forecasting as well as characterizing an en- 
during need. Predicting the precise form that technology will take at a future date is 
not reasonably to be expected. A "long-term" look helps to evaluate the probability 
and significance of possible future developments (circa 1990) so that present managers 
can make better decisions. 

The term "national" pertains to the United States and the aggregation of people 
having like institutions and customs and a sense of social homogeneity and mutual in- 
terest therein. Also implied is a national level of viewpoint rather than state or local, 
e.g. , national goals or national objectives. 

"Telecommunications" is communications at a distance. Here we mean a broad 
sense of any transmission, emission, or reception of signals, writing, images, 
sounds, or intelligence of any nature by wire, radio, visual or other electromagnetic 
systems. The terms telecommunication and communication are often used inter- 
changeably. 

"Needs and priorities" implies a hierarchy of imperatives for achievement. As 
posed here "need" is the central determinant of technological forecasts in the com- 
munications field. This expresses a teleological view that processes of technological 
changes in communications should be interpreted as being responsive to external 
stimuli rather than self -generating or "inner-directed" as in the ontological view. 
Further, such viewpoint tacitly assumes that technology is only utilized if it responds 
to a need. Recognizing that breakthroughs are essentially unpredictable, the central 
position of "needs" in forecasting relies on the expectation that major technological 
contributions in communications will result from applied, mission-oriented research. 
Statement of communications needs will strongly reflect the experience and expertise 
of the originator, which may be both a strength and a weakness. Interaction of social 
scientists and communications technologists are vital in developing needs. 
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In summary, the scope of this handbook is to explain the Delphi method and illus- 
trate its application in identifying communications experts and stating communications 
expectations which may be reflected in questionnaires. 

1.2.1 Context of the Delphi Method 

There are a number of forecasting tools that may be used to predict the tech- 
nology and hardware that will be of major concern in the future. Four tools are gen- 
erally described; Delphi fits into the last of these techniques: 

• Morphological analysis is a method for systematic examination of all possi- 
ble inventions, strategies, policies, or outcomes emphasizing fundamental 
structural differences and/or similarities rather than functional or perform- 
ance features. 

• Extrapolation of trends is the estimation of future values or magnitude by 
assuming smooth, continuous progression from present states or trends. 
Underlying causal mechanisms are not necessarily known. 

• Heuristic forecasts are based on explicit "models" of a dynamic process 
leading to or of the future, involving a greater degree of understanding of 
some causal mechanism than for a simple extrapolation. 

• Intuitive forecasts of future developments encompass flashes of genius, 
brainstorming, games, scenarios, consensus devices (Delphi), etc. Sys- 
tem and order may be applied to such sources to improve results. 

Most practitioners of forecasting agree that long-range forecasting is not merely an 
extension of short-r ang e forecasting. At the present time only intuitive techniques have 
been suitable for long-range forecasting. For this purpose, the Delphi method and 
scenarios have been most useful. When properly applied the Delphi technique will 
provide long-term insights to aid decisionmakers in formulating long-range plans. 
Coordinated information on the future in social, economic and technological areas is 
normally sought to indicate trends or patterns. Separate Delphi panels may be formed 
and addressed to cover all three areas. On the other hand, supplementary studies or 
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the application of other techniques may provide the additional information to be used in 
conjunction with a technological Delphi exercise. 

1.3 APPLICATION OF DELPHI TECHNIQUES TO LONG-TERM NATIONAL TELE- 
COMMUNICATIONS NEEDS AND PRIORITIES 

1.3.1 Application of Delphi . Delphi techniques can be used in any situation where 
opinions are sought from experts (or knowledgeable persons) on a particular subject or 
issue. This technique has been developed, tested, and found to be an extremely effec- 
tive method of forecasting future events in both business and government activities. 

Its success depends critically on the choice of experts to serve on the panels, the 
formulation of questions, and the way in which the technique is implemented. Delphi 
has been applied to a forecast survey of Japan's scientific and technological develop- 
ments up to 2000 A. D. , which included some space and communications programs (see 
the example, in Paragraph 6.1). 

1.3.2 Requirement for Long-Range Planning . As with other ongoing activities NASA 
can engage in long-range communications planning to identify the major decision points 
well in advance of their occurrence for the decisionmaker in order to prevent piece- 
meal decisions and to illuminate the long-term consequences of his decisions. In gen- 
eral, NASA programs span long time scales; communications developments face the 
same extended R&D cycle as the other programs. In this framework, long-range plan- 
ning is needed to provide communication programs that are viable alternatives to other 
programs - all within projected budgetary levels. Intelligent long-range planning re- 
quires insight into the social and military environments, as well as into the technology 
that will exist in the future. In fact, the practice of forecasting is deeply woven, ex- 
plicitly and implicitly, in our national system. 

1.3.3 Telecommunication Needs . ' A first step in NASA's long-range communications 
planning process is determination of long-term needs, from which concrete goals are 
derived for achievement. Forecasting technological advancements in communications 
is essential to this determination. Nearly everyone who looks into long-range fore- 
casting must face the fact that a large part of the activity is at least within the area of 
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opinion. The opinion of communications experts appears to be a likely source for indi- 
cating the features of the communications future. The Delphi technique is a useful way 
of eliciting and organizing this expert opinion. In recent literature much has been 
written about the revolution in communications, projections on the future of communi- 
cations, and communications needs. (Appendices B and C discuss anticipated develop- 
ments further. ) A caution is in order because even widely recognized needs do 
not necessarily call forth invention; a need is not always reflected in a market. 

1.3.4 Management of Delphi Application . In the application of Delphi techniques, the 
administrative/analytic group maintains constant control over the iteration of question- 
naires, processing of responses, and controlled feedback. Although this group must 
exercise extreme care to avoid introducing its own biases, such control is essential 

to reduce ineffectual results and incorrect actions based on low-quality forecast infor- 
mation. The final forecast information is assembled by the analytic group in usable 
form for making statistically valid decisions. This information can then be reviewed 
by higher level NASA management in order to make Suggestions on its validity, con- 
tent, and recommended usage. Finalized Delphi forecast information would then be 
submitted to top-level planners to use in conjunction with other decision data and aids 
in developing and issuing specific plans and strategies. 

1.3.5 Delphi Technique Should Avoid Gross Errors . Some experts are so immersed 
in the problems of developments in their field that they tend to be overly conservative 
on what may reasonably occur in the future. Delphi tends to stimulate and interest the 
participants by the feedback and iteration of questions, by the requests for justification 
of outlier opinions to the responses of the group of experts, and by permitting a 
respondent to change his mind. The outlier opinion may turn out to be the correct 
interpretation. Delphi should encourage the expression of insights and not be de- 
flationary to egos of panel members. 

3 

Arthur C. Clarke has collected a number of examples where the individual ex- 
pert has made a particularly bad forecast. A few of these are worth noting. 
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The first was an essay by the astronomer Simon Newcomb, who stated that "the 
demonstration that no possible combination of known substances, known forms of 
machinery, and known forms of force can be united in a practical machine by which 
man shall fly long distances through the air, seems to the writer as complete as it is 
possible for the demonstration of any physical fact to be." Newcomb was convinced 
(as were Euler, Stokes, Kirchhoff, and Rayleigh before him) that the physics of lift 
and drag on finite three-dimensional bodies moving in a viscous fluid (air) were such 
as to rule out any possibility of powered flight by heavier-than-air craft. Although the 
relevant physical laws were previously known in general, the correct deduction (i.e. , 
calculation) from them was not made until after a successful powered flight was actual- 

i 

ly demonstrated by the Wright brothers in 1903. 

A second instance was occasioned by the publication of a book called "Rockets 
through Space" by Cleator in 1936. A review of the book, written by R. v. d. R. 
Wooley and published in Nature, dismissed the notion of space flight as "essentially 
impractical." In 1956, a year before the first Sputnik, Dr. Wooley was appointed 
Astronomer Royal. When he was interviewed by the press he confirmed his earlier 
opinion with the remark, "Space travel is utter bilge." The author of this opinion be- 
came, ex officio, a member of the committee advising the government of the United 
Kingdom on space research. 

Aeronautical engineer Nevil Shute Norway (later famous as author Nevil Shute) 
was chief calculator for the R-100 Airship and a cofounder of Air Speed Ltd. , subse- 
quently merged with DeHavilland's in 1940. In 1929, he was very much an optimist 
about the future of civil aviation, yet he firmly predicted that by 1980 commercial air- 
craft would be limited to a cruising speed of 110 to 130 mph, a range of 600 miles, 

and a payload capacity of 4 tons out of 20 tons total weight. (On the other hand, in a 

4 

novel written in 1948, he accurately anticipated the problems of catastrophic struc- 
tural failure due to metal "fatigue" which later plagued the Comet and the Lockheed 
Electra, among other aircraft. ) 
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1.4 ORGANIZATION OF HANDBOOK 


The balance of this handbook is divided into six major sections and three appen- 
dices. 

• Section 2, Management of Delphi Procedure, describes the planning for im- 
plementation of Delphi techniques: the scheduling, technical control, and 
budgeting of a Delphi program. 

• Section 3, Delphi Methodology, presents the historical development of Delphi 
and definition of the basic method. An illustration of a Delphi application is 
given to involve the reader so that he may acquire an overview of the essen- 
tial steps. Finally, the principal advantages and limitations of the Delphi 
method are discussed. 

• Section 4, Procedure for Delphi Application to Long-Term National Telecom- 
munications Needs, sets forth an explanation of needs, how to select experts, 
how to generate pertinent questions, the processing and iteration of question- 
naires, associated statistical analyses, and presentation of results. 

• Section 5, Development of Delphi Modifications, describes techniques that 
various practitioners have evolved in applying Delphi methods with the aim 
of making results more accurate and valid. Several of these are associated 
techniques and not necessarily Delphi per se. 

• Section 6, Examples of Delphi Method Applications, presents four abstracts 
of Delphi studies, illustrating the flexibility, style, and usefulness of the 
method. 

• Section 7, Utilization of Delphi -Obtained Data, points out several systematic 
methods for evaluating the results of a Delphi effort as additional insights in 
the decisionmaking process. 

• Appendix A, Mathematics of Delphi Methods, provides details of mathemat- 
ical techniques useful in the Delphi process. 



• Appendix B, Summary of the Telecommunications Future, highlights the 
areas of expected developments in the communications field as reported in 
recent literature and as expressed in preliminary discussion with a few 
knowledgeable communications experts. 

• Appendix C, Literature Survey of Future Telecommunications Technology, 
discusses recent literature which provides additional perspective and 
previews of future telecommunications developments. 
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SECTION 2 - MANAGEMENT OF DELPHI PROCEDURE 


2. 1 PLANNING OF PROGRAM 

The many planning activities and questions that must be considered in the 
course of managing a Delphi exercise or a Delphi conference are outlined below, in 
the approximate order in which they arise in any given exercise. Many of the 
questions affect each other, however, so the ordering is somewhat arbitrary. 

5 

1. The exercise design and monitor team must be selected. Turoff recom- 
mends two to five professional people and about the same number of 
secretarial support people for a questionnaire-type exercise with 10 to 50 
people in the respondent group. The size of the design and monitor team 
is roughly proportional to the size of the respondent group. If a Delphi 
conference is used instead of a Delphi exercise, the number of secre- 
tarial support people could be drastically reduced. Throughout this 
document a Delphi exercise will generally imply the use of mimeographed 
or printed questionnaires distributed and returned by mail and tabulated 
manually; whereas, a Delphi conference will imply the use of interactive 
remote access computer terminals, such as teletypes, connected by 
phone to a central computer. The advantages and disadvantages of these 
two methods are presented in item 6 below. Further details on the Delphi 
conference are given in Paragraph 5. 6. 

In preparing a Delphi exercise or conference, there are three or four 
groups of people to be considered: the design and monitor team(s), which 
may be separate groups, or more usually, a single group; the respondent 
group; and the user group. The user group is the group with the 
questions, the sponsor of the endeavor. They, or some of them, may 
also be members of the respondent group, but Turoff says they should not 
act as monitors. However, he states the following as one of what he says 
are the only four positive statements that can be made about Delphi: 
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"The utility of the results depends upon the close cooperation between the 
design and the intended user body or at least a clear understanding by the 
design team of the goals or requirements of the user body*. " 

2. It follows, from the latter part of item 1. above, that either the user body 
or the design team must formulate, as clearly as possible, the purpose of 
the exercise. 

5 6 

Turoff lists the following possible objectives for a Delphi ’ : 

a. To determine or develop a range of possible alternatives. 

b. To explore or expose underlying assumptions or information leading 
to differing judgments. 

c. To seek out information which may generate a consensus of judgment 
on the part of the respondent group. 

d. To correlate informed judgments on a topic spanning a wide range of 
disciplines. 

e. To educate the respondent group as to the diverse and interrelated 
aspects of the topic. 

f. Examining the significance of historical events. 

g. Gathering current and historical data. 

h. Putting together the structure of a model. 

i. Delineating the pros and cons associated with potential decision or 
policy options. 

j. Developing causal relationships in complex economic or social 
phenomena. 

k. Clarifying human interactions through role playing concepts. 

In addition to being influenced by the purpose of the exercise, the questions 
and format of the Delphi and the number and selection of respondents may 
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be influenced by whether or not the results are to be confidential or 

widely published, and whether the Delphi is intended as a supplement, 

precursor, or alternative to a conference or committee meeting. In 

Reference 5, Turoff states his view that the proper role of a Delphi dealing 

with policy matters is as a precursor for committee activity. At least in 

the factual field, however, it has been shown that following up a Delphi 

exercise with a face-to-face conference usually degrades, rather than 

7 

improves, the accuracy of the results produced by the Delphi. There 
is, therefore, some question as to whether it might not be better to use 
a Delphi as a replacement for a committee discussion, rather than as a 
precursor to it. 

3. The number of respondents, and the criteria for their selection, is the 
next topic which must be considered. It has been shown that results 
improve with the size of the group, at least up to 25 or 30, although the 
difference between groups of 13 and 30 is not very large. One must allow 
for the chance that some of the respondents will drop out, for one reason 
or another, after the exercise begins. Furthermore, if one wants to base 
the results on the answers of self-rated experts, one needs a larger group 
to begin with, so that the group of people who rate themselves as expert on 
any particular question will generally not be too small. 

Budgetary limitations restrict the maximum size of the respondent team 
which is practical, particularly if they have to be paid to participate. 

Even if they are not paid, the more respondents there are the more com- 
puter or clerical time will be required to process their answers, the more 
time will be required from the monitor team to secure their cooperation 
and answer their questions, etc. 

The criteria for selection of the respondents is very important. A second 
of Turoff's four positive statements is: "Success of the Delphi is dependent 
upon the ingenuity of the design team and the background of the respondent 
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group. " (His other two statements are: "No hard and fast rules exist to 
guide the design of a particular Delphi" and "The Delphi requires a degree 
of quantification to be imposed upon subjective judgmental factors and the 
definition of this quantification is a matter of principal concern to the 
design team. ") The two major factors that must be included in the set of 
criteria for respondent selection are expertise and diversity. Obviously, 
it is desirable to select as respondents people who know something about 
the subject in question. It is also very important to get an interdisciplinary 
approach. In a Delphi on communication needs, for example, it would be 
desirable to have respondents who are in the business of providing com- 
munications, respondents who are users of these services, some who are 
on government regulatory bodies, and some with an academic background 
in the subject. Each of these, in turn, could include diversity within itself. 
For example, the users could include broadcasters, amateurs, the mili- 
tary, users of police and taxi bands, aircraft and marine users, radio 
astronomers, etc. Paragraph 4. 1 gives further details on the selection of 
communications panel members. 

After the number of respondents and the criteria for their selection have 
been decided upon, there are the tasks deciding who will nominate can- 
didates for the panel, making the nominations, choosing among them, 
contacting them and persuading them to participate, making the financial 
arrangements if they are to be paid, etc. If the user group participates in 
the nominations or selection they may unconsciously bias the panel with a 
preponderance of people who see things the way the user group does. 

4. The question of whether or not the respondents will be paid must be con- 
sidered. If they will be paid, more questions can be asked, more types of 
information can be requested with each answer, and more rounds can be 
run without the respondents losing interest and dropping out. Although a 
Delphi exercise or conference takes less of the respondent's time than a 
conventional conference would, . each round may still take an hour or more, 


2-4 


particularly if the respondent must fill out cross-impact matrices, or if he 
is asked to put next to each answer his rating of his own expertise on that 
question, his assessment of the importance of the question, and various 
other types of supplemental information which will be mentioned below. 
Furthermore, respondents will take the exercise more seriously if they 
are involved as part of their normal job function or if they are paid a 
normal consulting fee than if they are asked to volunteer their time. 

5. The amount of anonymity must be decided by the design team. Should the 
participants know the names of the other participants, and their back- 
grounds? Their backgrounds but not their names? Neither? If a partici- 
pant wants to identify himself by name when making a comment should he 
be permitted to do so? (The voting is always strictly anonymous, in a 
Delphi exercise, but strict anonymity may not be required for comments. ) 

Several authors have stressed that it is important that the panel of 
respondents know they are dealing with their peers. It may be desirable 
to let each participant know the names and backgrounds of the other par- 
ticipants. However, in a number of Delphi studies anonymity of respond- 
ents has been strictly preserved. 

6. As has been mentioned, an important decision is whether to conduct the 
Delphi by mail, using questionnaires, or by telephone, using interactive 
terminals such as teletypes, possibly supplemented with mailed hard-copy 
summaries in the case of Delphi conferences which last many weeks. 
(Summaries could also be produced on high speed line printers, if the 
respondents have access to such facilities. ) 

The computerized Delphi conference has the following advantages over the 
Delphi exercise using mailed questionnaires: 

a. It is usually faster than the mailed questionnaire procedure. Some 
Delphi conferences, however, go on for weeks or months. 
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b. It is usually less expensive to process responses by computer 
than to process them manually. The mailed questionnaires 
could be converted to punched cards and processed by machine, 
however. 

c. If the Delphi conference is kept short (1 day to 3 days) relatively little 
"rethink" time will be required from the participants. With the mailed 
questionnaires, so much time can elapse between rounds that the par- 
ticipants may have to start thinking about each question all over again 
on each round. 

d. Delphi conference computer programs can contain interactive features 
which are useful to the respondents. For example, the computer can 
help them eliminate inconsistencies in their probability estimates, can 
run Monte Carlo solutions of cross-impact matrices, etc. 

e. As Turoff says in Reference 6, "A significant observable effect of a 
computerized conference system is the group pressure to restrict 
discussion to the meat of the issue. Verbose statements always tend 
to receive low acceptance votes and individuals quickly learn, because 
of this, to sharpen their position if they wish to make a point. " 

The disadvantages of the computerized approach are: 

a. There is some inconvenience, even though the computer programs, 
running such Delphi conferences are designed to be as convenient as 
possible. Teletypes are noisy and fairly slow in printing. The limited 
width of teletype paper makes it hard to display tables with many 
columns. Line errors, computer malfunctions, and other problems 
slow things down. Some of these problems can be alleviated somewhat 
if the respondent lets his secretary or data technician handle the 
terminal. Such a procedure, however, reduces the usefulness of the 
interactive features of some Delphi conference computer programs. 
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b. Some of the respondents may not have access to teletype terminals. 
Terminals which operate more quietly and at a higher speed than 
teletype are available, reducing the impact of disadvantage (a), but 

I 

even fewer respondents are likely to have access to them. Of course, 
access to a terminal could be made one of the criteria in choosing 
respondents, but that might eliminate some respondents whose other 
qualifications were superior. 

c. In order to use the computerized procedure, one needs access to a 
computer and a computer program. Several programs are available, 
as mentioned in Paragraph 5. 6, but they run only on certain computers. 
This is not really a serious objection, since there are a multitude of 
time-sharing computer services that provide nationwide access to 
their computers, and some of these use Univac 1108 computers, 

which were used for Turoff's Delphi computer program. Further- 
more, programs can be modified to fit other computers, but this does 
involve costs (possibly a few man-months of effort, depending on the 
computer and the program). 

d. The overall cost of the Delphi conference will probably be a little 
higher than the cost of a Delphi exercise, because of the cost of 
computer and terminal time. Some of the respondents may also 
have long distance line charges, if they are not near a concentrator 
of the time-sharing service being used. 

Among the advantages of both of the above types of Delphi over conven- 
tional conferences is the savings in travel time and expense. There is 
also a gain in convenience to the respondents, since they can answer 
when they can spare the time (within limits), rather than having to 
reserve a particular date for a conference. If the design team holds 
the respondents to a very narrow schedule, such as one round per day, 
this advantage is somewhat reduced, but at least the respondent can 
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answer at the time of the day he wants, which is especially handy if he 
is in a distant time zone. 

7. Scheduling the overall project is the next problem. This schedule must 
include the time for the design team to complete the consideration of the 
topics below, contact the respondents, and get their consent; decision on 
the number of rounds to use and the time the respondents are given to 
answer each round; and, for the exercise conducted by mail, the time to 
allow for mail deliveries both ways plus processing and tabulation of the 
results of each round. 

At least four rounds are generally used. On the first round, the respond- 
ents give their initial estimates. On the second round, those who find 
themselves in the bottom or top quartile, and who do not want to move into 
the interquartile range, are asked to give their reasons. These reasons , 
are displayed to the other respondents in the third round, at which time 
other respondents may refute them. Based on these refutations, which are 
displayed in the fourth round, some respondents may change their votes. 

On the other hand, some new reasons or refutations may be presented, 
suggesting that it may be fruitful to go on to a fifth or sixth round. Such a 
situation is especially likely if policy questions are being considered. 

Turoff notes, in regard to policy Delphi studies, "On some issues where 
strong polarization existed in the respondent group, we discovered that 
each side on obtaining the summary of the first round did not really believe 
that the other held so completely different a view and felt that a few casual 
comments would shift the other group to their "logical" position. Upon 
getting the results of the second round this belief is usually shattered. 

When the positions were still far apart on the third round, some respondents 
in the past Delphi exercises have become disgusted and given up on the study, 
however, a good many began to work very hard at developing very careful 
arguments and comments. In essence, it was not until the third round that 
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the majority of the respondents reacted to the seriously polarized issues. " 

In answering technical questions, on the other hand, two or three rounds 
may be sufficient, if there are few reasons given and little changing of 
votes. 

The number of rounds is less distinct in a computerized Delphi than in one 
run by mail. In the computerized version, a few people may interact with 
each other online or at short intervals, having many rounds, while others 
may be off-line for days at a time. One participant may thus give his fifth 
round refutation before another gives his second round estimate. 

As has been mentioned, allowing the participants a short time for each 
round of questions reduces the rethink time required, and reduces the 
overall time required for the Delphi, but also reduces the convenience, 
since some participants may be busy with something else for a few days. 

8. The next step is to decide on the format of the questions, the approximate 
number of questions, the general subject of each question, and finally the 
specific wording of each question. Paragraph 4. 2 discusses the generation 
of the questions in detail. Here it is sufficient to note that virtually every 
Delphi exercise has used a different format for the questions and answers. 
Some ask for dates, some ask for probabilities. Some ask for self- 
assessments of expertise on a question as compared with the other partici- 
pants, some ask for self-assessments on a particular question as compared 
to the other questions, and some don't ask for self-assessment. Some ask 
for cross-impact matrices, others do not. Those asking for cross-impact 
matrices use many different approaches, as described in Paragraph 5.8 and 
Appendix A. 3. The respondents may be asked to rate comments or proposed 
items for importance, validity, desirability, and/or feasibility or probability 
of occurring, as applicable. 

If a computerized Delphi conference is planned, and the desired format 
doesn't agree with the format provided in available computer programs, 
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some time will have to be allowed for modifying the program. This should 
not be too difficult if the program is in a high level language such as 
FORTRAN, BASIC, XBASIC, or JOSS. 

The design team must also decide how much freedom should be given to the 
respondent group to change the nature of the issues presented, and should 
set guidelines on the editing of comments generated by the respondents 
and fed back without editing. At the other extreme, the monitor group can 
consolidate and summarize generally similar comments from many partici- 
pants; perhaps edit out emotional phrases and arguments, if they think that 
desirable (it may not be); and inject arguments and rebuttals of their own, 
if they think the participants are overlooking or misinterpreting something. 
If they inject comments of their own, they must decide whether to identify 
such comments as coming from them, rather than from the respondent 
group. 

In view of all the above decisions which must be made, ample time must be 
allowed for the formulation of formats and questions, plus some time for 
trying them out on a pilot sample of people outside the design team, and 
revising them to increase clarity, based on the results of the pilot test. 

The pilot run should also test the clarity of the instructions to the respon- 
dents, discussed in 9 below. 

9. Closely associated with the detailed formulation of the questions is the 

formulation of the instructions to the respondents. Decisions must be made 
on whether a neutral judgment will be considered the same as a no judg- 
ment or blank answer; how many permissible answers are allowed to each 
question, and whether a neutral judgment is a permissible answer. For 
example, in Reference 5 no neutral answer was permitted. The respondent 
had to feel that an item was either a little important or a little unimportant. 
A no judgment category was, however, allowed. The set of instructions 
presented in Reference 5 are particularly clear. They were designed for a 
policy Delphi, however, and would require some revision for other types. 
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Many other specific decisions must be made in formulating the instructions. 
In a computerized Delphi conference, a decision must be made as to how 
many responses must be received before the computer summarizes the 
votes, computes the new median and quartile values, and makes these 
figures available to respondents who subsequently log in. A decision must 
be made on exactly what information is fed back. For example, is the 
number of respondents who answered a question (not counting the "no 
judgments") displayed? Are the results of both the overall group and the 
self-rated experts displayed, or only one of these? What are the exact 
criteria for adding an item to the list to be considered, or dropping an 
item from the list? Can the author of a new item later modify its wording? 
Can someone else later restore the original wording? Should there be a 
limit on the number of active items? On the total number of active and 
inactive items? If so, what should these limits be? How does one feed 
back a summary of group response on non-numeric items which have no 
median (for example, the answers to questions like "In what ways will 
communications in the year 2000 differ from the year 1974? "). Most of 
these decisions must be made in running a Delphi exercise as well as in 
r unnin g a conference, but the decisions must be more explicit in the 
conference situation if they are to be programmed into a computer. 

Turoff says "The design effort can take a month, and possibly two or three 

months if no member of the design group has a strong background in the 

5 

subject. Outside consultants can be extremely useful in this phase." 

He also says that the design team should prepare a fact and data summary 
indicating whatever appropriate benefit or cost measures are available on 
a historical basis. This summary would be supplied to the respondents. 

His remark was in connection with a policy Delphi. For other types of 
Delphi, other historical information should also be supplied to the respond- 
ents. Such would probably include historical trend data on usage of 
various types of equipment and communication bands, in a communication 
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study. Other data might include graphs relating telephone traffic to per 
capita GNP and total GNP, data relating telegraph, telephone, and teletype 
traffic to various trade statistics, etc. 

10. Finally, the plan must allow some time for the preparation of a final report 
summarizing the findings of the exercise or conference, the procedure 
used, etc. 

2.2 SCHEDULE REPORTING AND CONTROL 

The time allowed for the respondents to complete their answers to each round 
of questions, and the time the monitor team takes to process and tabulate their 
answers and issue the next questionnaire (in a manually processed Delphi exercise) 
are in the critical path of the exercise. Accordingly, if the results of the exercise 
are wanted in a relatively short time (as is usually the case), it is important to hold 
these times on each round to a minimum. 

In Reference 5, Turoff describes a Delphi exercise in which the participants 
were requested to mail their answers to each questionnaire within 3 days of receiving 
it. He stated that the results of a round can be manually analyzed and summarized in 
1 or 2 weeks with allowance for overtime when required. He adds, ’’The processing 
of the returns does not lend itself to a normal eight-hour day, five-day week schedule, 
and the peaks in professional and secretarial work load do not coincide. Because of 
the complexity of the issues, it is probably preferable for a professional on the 
monitor team to plan to work on a given well-defined section of the exercise in one 
sitting and to ensure that another professional will double check him before the sum- 
mary and new questionnaire are prepared. " 

In addition to the above times, one would have to allow several days each way 
for mail delivery, if the respondents are spread over a large geographic area. 

The above figures were for a policy Delphi he conducted which started with 
70 items. In this exercise 295 items were generated by the respondents and fed 
back for. evaluation, and 74 additional items of interest were not fed back but were 
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summarized in an appendix to the report. Many of these comments were either 
emotional or fell in the sections from which he excluded feedback. On the last round, 
a free form question asking for respondent's final recommendations generated 46 
items which were compiled separately but tended to duplicate much of the earlier 
material. In addition, approximately 100 items were purged completely as outright 
duplication or not meaningful (as judged by the monitors). Therefore, the original 70 
items generated aboue 500 items on the part of the respondents, about a 7:1 ratio on 
the average. The more controversial items generated the most new items. 

In a manual operation of this type the time required to process the returns can 
be expected to be proportional to the number of respondents and to the number of 
items, and inversely proportional to the size of the staff. 

It is possible to shorten the duration of the Delphi dramatically by using the 
computerized approach discussed in Paragraphs 2. 1 and 5. 6. In such an approach 
the mail delivery time and the time required for analyzing and summarizing each 
round are eliminated. In Reference 8, Turoff describes a computerized Delphi 
conference which ran for 13 weeks. In that case the convenience of the respondents, 
rather than a sense of haste, was the factor considered most heavily in establishing 
the schedule. One round per day could be maintained, if time was urgent, the scope 
of the study not too extensive, and the list of questions not excessively long. At this 
pace, relatively little editing of the new items could be expected from the monitor 
team. 

Reference 8 included a report on an experiment to determine the effects of 
time permitted to answer each question on the accuracy of the answers. The results 
showed that accuracy was best if the respondent spent only 30 seconds on each 
question. Fifteen seconds was too short a time for the respondents to take all factors 
into account, but times longer than 30 seconds were also not as good from an accuracy 
point of view. Although the test was on almanac-type questions, the respondents had 
to make fairly complicated judgments. For example, a question like, "What was the 
popular vote for Kennedy in the 1960 presidential election in the state of Texas? " 
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involves a number of factors: the population of Texas in 1960, the fact that Texas is 
a southern state, predominantly Democratic, but conservative, and predominantly 
Protestant, that Kennedy was a Catholic, and so on. As Dalkey says, "Apparently, 
there is a fairly sharp limit on the number of factors and the amount of 'processing' 
that can be dealt with profitably. At all events, we seem to have validated the advice 
frequently given in connection with objective examinations - 'Trust your first 
estimate'. " 

If the respondents are told to spend only 30 seconds on each answer, it should 
not take them very long to complete each round. 

Regardless of whether the Delphi is conducted by mailed questionnaires or by 
computer, it is desirable to have a procedure to check on the timely receipt of 
answers from all respondents, and to flag respondents who are late in answering. 

The monitors should contact such respondents by telephone, to find out if they are 
having difficulty, and to try to alleviate whatever problem is causing the delay. 

If the exercise is conducted by mail, it is possible to speed up the tabulation 
of the replies if they are punched into cards and processed by machine. Machine 
processing is also more accurate and cheaper than hand tabulation, once the program 
has been written or otherwise obtained. 

2.3 TECHNICAL REVIEWS 

It has been shown that the interquartile values Q and Q (defined in Appendix 
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A. 2) of the answers to each question generally move toward the median as the Delphi 
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progresses through its rounds. Such a movement indicates and defines a growing 
consensus among the respondents. Accordingly, the monitor team should maintain 
a close watch on the changes in the interquartile range from round to round for each 
question. If some questions show little change, or a divergence in the interquartile 
values as the Delphi progresses, something may be wrong. Perhaps the phraseology 
of the question is unclear, and different participants are interpreting the question in 
different ways. In such a case it would be worthwhile for the monitor team to look 
closely at the questions and comments responsible for such behavior, and perhaps to 
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talk to some of the people who have changed their votes away from the median, to find 
out what their interpretation of the question is and the reason for their change of vote. 

If a computerized Delphi conference is being run, it would be desirable to check 
by telephone with some of the respondents, to get from them information on difficulties 
or undesirable system characteristics they have encountered, and ideas for improve- 
ments in the system. Alternatively, one of the items given to the respondents might 
be a request for their opinion of the system, a list of defects they found, and sugges- 
tions for improvements. 

If the respondents are asked to construct a cross-impact matrix, the matrix 
which results from their consensus can be used as input to a Monte Carlo program 
after each round of the Delphi, and the results fed back to the participants, so they 
can change their probability and conditional probability estimates if they feel the 
results of the Monte Carlo are unrealistic. The Monte Carlo results of their indi- 
vidual cross-impact matrix should also be fed back to them. The computer can also 
check the cross-impact matrix and their probability estimates for consistency with 
equations of the form of equations (1) and (2) in Appendix A. 3, and bring any incon- 
sistencies to the attention of the monitors. If the monitors feel the inconsistencies 
are large enough to be worth passing on to the respondents, they should do so, and 
the respondents may then improve their estimates during the next round. 

2. 4 BUDGETING FOR PROGRAM EXECUTION 

It is convenient to divide the costs associated with a Delphi exercise or confer- 
ence into three parts: design, execution, and report. 

As has been mentioned, Turoff recommended a team of two to five professionals, 
and the same number of secretarial/clerical people, for a manual Delphi exercise with 
10 to 50 respondents and 70 initial items on the questionnaires. 

The design effort can be expected to be approximately proportional to the 
number of items on the questionnaires. The design effort will also depend somewhat 
on the number of people in the respondent group, since they all have to be chosen. 
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contacted, given some instruction, etc. Furthermore, if a large respondent group is 
to be used, it is appropriate to put more effort into the design of the questions and 
instructions. 

If a computerized Delphi conference is to be run, rather than a mailed question- 
naire type of exercise, the design effort could be expected to be somewhat larger. 

All of the things which have to be done during a mailed exercise also have to be done 
with a computerized conference including the payment of consultant fees or other 
renumeration to respondents. In addition, one must decide which computer to use; 
which computer program to use; whether to modify the program, and, if so, how; 
make sure the respondents have access to terminals compatible with the computer 
and program, etc. 

If the computerized conference approach is selected, there will probably be 
some cost associated with obtaining the necessary computer program. This cost may 
be rather nominal. For example, we understand that Turoff's program is available 
for $300. Other programs, of course, may be more or less expensive than this. 

Any program that is obtained will probably have to be modified to fit a particular 
conference format, unless the format was planned with the program in mind. 

The difference in effort between the mailed questionnaire and the computerized 
approach may not be quite as large as the above paragraph would imply, because 
before a decision is made on which approach to use, the design team will probably 
'want to review the formats and instructions associated with available computer pro- 
grams, to see if they are suitable. The effort of obtaining and comparing this infor- 
mation would therefore be incurred whichever decision is finally made. 

A computer program which compiles and tabulates the replies to a mailed 
questionnaire can be expected to be simpler and cheaper than one which is used to 
run an interactive, computerized Delphi conference. Nevertheless, such a program 
can be a fairly expensive item itself. In Reference 10 it is mentioned that the 
respondent's questionnaires were converted to 40, 000 IBM cards which were then 
manipulated by over 30 separate computer programs written primarily for that 
project and using a variety of separate computing systems. 
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That was an effort involving 210 respondents and 500 multiple-part and graphi- 
cal questions. 

During the execution phase, Turoff reports that each round of the exercise can 
be manually analyzed and summarized in 1 to 2 weeks with allowance for overtime 
hours when required. He states that the secretarial work load, which is compara- 
tively massive, is not well distributed time-wise during the redesign period between 
rounds. This could be cut to some extent by the use of online text editing and com- 
position systems now available on a number of time-shared computer systems. The 
peaks of the professional and secretarial work load do not coincide. These figures 
were for the aforementioned policy Delphi exercise he ran. 

In a computerized Delphi conference the secretarial work load during the execu- 
tion phase of the project would be virtually eliminated. The exercise can be designed 
so that relatively little editing by the monitors is required. For example, rather than 
the monitors deciding on which comments to consolidate, which to eliminate, and 
which to modify, all comments can be fed back unedited to the respondents, and can 
be dropped by the respondents if a certain fraction vote to do so. If that is done, the 
monitor team would have a much smaller work load during the execution phase. They 
would mainly answer questions, clarify instructions, and perhaps modify the computer 
program or instructions if the respondents so request. However, the editing task 
would add to the work load of the respondents. 

In a computerized Delphi conference that Turoff ran, 20 respondents used 100 
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hours of terminal time over a period of 13 weeks. Central processing time on a 
Univac 1108 was about 1 hour. Less than 100, 000 characters of storage were used 
in this exercise, which was limited to 99 active plus rejected items. Initially, 
respondents were on the terminal for about a half hour per interaction. Once the 
novelty of the terminal and the exercise wore off, 10 minutes was a more common 
interaction time for the purpose of seeing new discussion items and adding or 
changing some votes. Respondents requested, and were given, three hard-copy 
summaries at 3 to 4 week intervals. By the time the conference was over, at least 


2-17 



10 of the respondents were using secretaries or junior staff to obtain the latest items 
and submit responses as directed. Turoff states, "The success of this alternative is 
extremely important in obtaining the participation of any busy individual. Being able 
to treat the system as one would treat the news wire service is probably beneficial to 
its potential application. ” 

In this connection, it should be noted that it is important that the computer 
program used have an option available to permit the user to receive detailed instruc- 
tions the first few times he interacts with the system, or to receive very abbreviated 
cues, once he is familiar with the system. Turoff's program had this feature. The 
junior staff, however, generally preferred to continue to use the long, slow, detailed 
option. 

Turoff estimated that the conferees spent one additional hour thinking about the 
issues for every hour on the terminal. Thus, the average respondent spent 5 hours 
away from the terminal, and he or his assistant spent 5 hours at the terminal during 
this exercise. 

In estimating the costs of an exercise of this kind, one must estimate the 
charges for terminal time, computer time, and computer storage. Different time- 
sharing services have different rates, and even a single service usually has a multi- 
plicity of alternative rate plans the customer can choose among, but the following 
rates are believed to be fairly typical: 

Terminal time: $12. 00 per hour during prime time (7 AM to 7 PM) for nation- 
wide access to the same computer. Regional access would be about $11. 00 per hour. 

Central processing unit time: 50£ per second on a Univac 1108 

Online storage on disc: 2.5£ per day per page of 2, 098 characters 

If a respondent is not located in one of the cities that provide entry to the time- 
sharing system, he would have to include long distance line charges to the nearest 
entry point in addition to the terminal time costs. 
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The above charges do not include costs for the rental of the teletype or other 
terminal equipment, which are usually treated as overhead at most installations. 
Presumably they are available anyway, so there is no marginal cost. If the ter- 
minals at a particular location are heavily used, there may be some cost in the form 
of delay to others who wish to use the terminal for other work. 

If the Delphi were run by mail and the results tallied by computer, computer 
costs would be much lower. The computer time would be considerably shorter if the 
tabulation were done in a batch mode rather than in an interactive mode, and the 
charges per hour would be much lower if the runs were made during off hours. At 
the lowest priority, the charge for CPU time drops to about 10 £ per second. 

Running the exercise by mail and tabulating the votes by computer would require 
an allowance for some clerical time in converting the respondent's replies to punched 
cards. On the other hand, there would be a corresponding reduction, or perhaps a 
larger reduction, in the time required by the respondents to obtain instructions and 
data from the terminals and enter their answers. 

About the only cost item which would be larger for mailed questionnaires than for 
a computerized Delphi conference in addition to the processing of responses is the 
rethink time required of the participants, if the exercise stretches over a long period 
of time than the conference. 

In summary, the overall expense of the Delphi exercise using mailed question- 
naires should be less than the expense of a computerized Delphi exercise. The dif- 
ference may not amount to a great many dollars, however, and the advantages of the 
conference over the exercise may make it a better choice in spite of its somewhat 
higher cost. 

Both the conference and the exercise would probably be cheaper than a face-to- 
face conference if some of the respondents would incur large travel charges in going 
to a conference site. The computer charges above come to about $3000. If 20 con- 
sultants had to travel to some city for a face-to-face conference, their travel and 
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lodging bills alone might total more than $3000 if they were not located close to the 
conference site, and they would have to spend many hours in travel in addition to the 
time required for the conference. The time required for the conference itself would 
be considerably more than the time required to fill out the Delphi questionnaires. 

It is reasonable to believe that expenses during the execution phase of the Delphi 
are proportional to the number of respondents and to the number of items they are 
asked their opinions on. 

At the end of the exercise, the monitor team will require some time to write up 
the results of the exercise or conference and publish a report. This effort will proba- 
bly be approximately proportional to the number of items in the Delphi, and will be 
affected somewhat by the number of respondents, since the more respondents there 
are, the more comments and new items are likely. 
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SECTION 3 - DELPHI METHODOLOGY 


3.1 DEFINITION OF DELPHI AND HISTORICAL DEVELOPMENT 

Delphi methodology is essentially the repeated interrogation of a panel of experts 
to obtain a systematic consensus of informed opinions. As such, Delphi solicits in- 
formed judgments on what will happen and when in a stream of events it will happen. 

The members of the panel are queried separately in order to avoid bandwagon effects 
and authority biases. After the members of the panel submit their estimates, a median 
of the responses is computed with an interquartile range and outerquartile range (see 
Figure 3-1). 

OUTERQUARTILE OUTERQUARTILE 

RANGE RANGE 

II I 


INTERQUARTILE RANGE 
(50% OF JUDGMENTS) 



FIRST SECOND THIRD 

QUARTILE QUARTILE QUARTILE 

(MEDIAN) 

Figure 3-1. Range of Responses 

Justification statements are requested from members of the panel whose judg- 
ments were in the outerquartile range. These statements and the statistical distribu- 
tion of the responses are retransmitted to the members of the panel who may then alter 
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their judgments or repeat them. At least three rounds of questionnaires are usually 
required to obtain a "homing in" or convergence of a group of opinions. 

The Delphi method evolved from a series of pioneering efforts at RAND Corpora- 
tion under the direction of Dr. Norman Dalkey and Dr. Olaf Helmer. Their milestone 
paper "An Experimental Application of the Delphi Method to the Use of Experts" 1 de- 
scribed the philosophical basis of the Delphi system for combining expert opinion into 
a group consensus. This basis uses what can be called, "cybernetic arbitration;" 
cybernetic because the process of deliberation is steered through feedback, by a con- 
trol group. 

Delphi is the name of a Greek town and is famous, in Greek history, for the 
oracle who resided there. Ancient civilizations looked for signs to help them make 
important decisions, and consulting oracles was one of the decisionmaker's tools in 
those days. The name Delphi was proposed by the RAND group to suggest the practices 
of ancient Greeks who obtained the counsel of a deity concerning the future through 
questions submitted to the Delphic Oracle, who was the most renowned of all oracles. 
This particular oracle was chosen because the RAND group first envisioned that this 
method could be used for better political forecasts as the Delphic Oracle had so often 
been utilized by the ancient Greeks. Indeed, the oracle at Delphi was consulted before 
any important step was taken in affairs of state. 

The Delphi procedure consists of seeking a consensus of opinions among experts 
about a particular subject and about conditions that will prevail in the future, thus 
explaining its increasing use in forecasting. 

The main features of the Delphi method are: 

1. No member of the panel of experts knows what another member of the panel 
says about a particular question. However, this does not mean that the 
identity of the other members is not known. 

2. Questions are asked in two or more iterations. They may be the same on 
each round and thus force a reexamination of views from the members of the 
panel; or they may consist of new questions raised by earlier responses. 


3-2 



3. At each iteration, additional information is given to the participants, in the 
form of statistics on the earlier responses of the group as a whole, or the 
answers and comments themselves. This feedback mechanism might also 
include additional data obtained outside the group. 

The Delphi method is thus an important attempt to bring rigor and logic to fore- 
casting by obtaining the consensus of a number of experts in a climate that eliminates 
the influence of personalities. Strictly speaking the Delphi method is not a forecasting 
technique, but a means for obtaining a consensus. 

After its initial impetus from Helmer, the Delphi method has been extensively 
used to determine long-range forecasts of expected technological and sociological 
developments. 

The 1964 Report on a Long-Range Forecasting Study by T. J. Gordon and Olaf 
2 

Helmer shows the application of the Delphi method to scientific breakthroughs in 
physical and biological technologies, world population growth, innovations in auto- 
mation, progress in space, new weapon systems, and the causes and the prevention 
of wars. 

A forecast of computer developments and applications was performed by Parsons 

and Williams in 1968. 11 George B. Bernstein prepared a 15-year forecast of Informa- 

12 

tion Processing Technology for the U. S. Navy in 1968. A. Douglas Bender of Smith, 
Kline, and French Laboratories conducted a set of Delphi studies on various aspects of 
the future of medical care, in the late sixties (unpublished report). 

TRW, Inc. adapted the Delphi method to their long-range planning and have con- 
ducted a series of long-range forecasts between 1966 and 1970 designed to show the 

13 

adaptability of the Delphi method in an industrial environment (see Paragraph 6.4). 

The Delphi method has been applied in medicine, in 1971-1972, to the problem of 

14 

extrapolating the results of animal experiments to human equivalents. 
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A detailed survey exploring the views of American economicsts on economic, 

social, and political trends throughout the rest of the century has been conducted by 

15 

the Delphi method, in 1972-1973. 

The Japanese Government has used the Delphi method to foresee socio-economic 

changes so as to formulate a science and technology policy. This forecast covered 30 

16 

years up to the year 2000 and was performed in 1970 (see Paragraph 6.1). 

The applications of the Delphi method are therefore numerous and cover a large 
range of activities. In every case, the method has been tailored to each application 
and some innovation has been introduced to improve on past results. Examples of such 
innovations are: (1) having respondents rate themselves relative to their degree of 
expertise on the subject matter, and (2) establishing an elite group of experts from the 
population of respondents. 

3.2 ILLUSTRATION OF DELPHI APPLICATION 

A simplified example is included in this section in order to provide the reader 

with a quick overview of the essential steps in the Delphi method. This example was 

used in an audience participation workshop held at a Conference on Technological Fore- 

17 

casting and discussed in "The Delphi Method - An Illustration" by Dr. O. Helmer and 
Mr. T. J. Gordon and is reprinted here by permission of Prentice-Hall Publishing 
Company. The questions encompass a wide spectrum of subjects. Summaries of 
results and group consensus charts are at the end of the article. It is strongly recom- 
mended that the reader should: 

1. Answer each question. 

2. Have a number of his associates respond to these same questions and then 
he should conduct a feedback as required in Delphi methodology. 

This is the fastest way to familiarize oneself with the workings of Delphi: a learn-by- 
doing approach. It is essential for a Delphi study leader to acquire this familiarization 
with integration of group opinion in order to effectively perform such a study. 
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The Delphi Method — 
An Illustration 


One way to understand the concepts, potential, and limitations of a technological 
forecasting method is to try it. Dr. Olaf Helmer, assisted by Mr. T. J. Gordon, 
conducted an interesting audience participation session at the Conference on Tech- 
nological Forecasting. Although limited by time and selection, this simple illustra- 
tion quickly and effectively familiarized participants with the attributes and theory 
of the Delphi method. 

The following example is presented as an illustration of the Delphi 
method. It describes a Delphi session which was conducted by Dr. Olaf 
Helmer and was evaluated by Dr. Helmer and Mr. T. J. Gordon at the First 
Annual Technology and Management Conference. The session was intended 
to give conference delegates an experience in the Delphi technique, and 
the participants were selected at random, not on the basis of their expertise 
in regard to the session's subject matter. Consequently, attention should be 
directed toward the methodology, not the results. Under ideal conditions, 
each of the participants would have been selected for his particular knowl- 
edge of the fields for which projections were being made. 

STEP 1 


Questionnaire 1 (see Exhibit 1) was distributed to over one hundred 
conference participants and each was asked to complete the form and return 
it to Dr. Helmer. In addition to giving his answers, each participant was 
asked to rank his expertise to deal with the particular questions. A "one” 


113 


3-5 


was to be placed in the box beside that question which the participant felt 
most competent to answer, anti a “seven” alongside the question about 
which he believed hintself to be least competent. Each of the remaining 
questions was to be ranked so that every number from 1 to 7 was used ex- 
actly once. Although questions were raised concerning biases in the wording 
of the questions, for purposes of the illustration participants were asked to 
accept the questions exactly as they were presented on the form. 


EXHIBIT 1. 

QUESTIONNAIRE #1 

Thii it the first in a series of four questionnaires intended to demonstrate the use of the Delphi 
Technique in obtaining reasoned opinions from a group of respondents. 

Each of the following six questions is concerned with developments in the United States within 
the next few decades. 

In addition to giving your answer to each question, you are also being asked to rank the ques- 
tions from 1 to 7. Here "1" means that in comparing your own ability to answer this question 
with who* you expect the ability of the other participants to be, you feel that you have the 
relatively best chance of coming closer to the truth than most of the others, while a "7" means 
that you regard that chance os relatively least. 

QUESTION ANSWER * 

1. In your opinion, in what year will the median family income (in I - 

1967 dollars) reach twice its present amount? 

2. In what year will the percentage of electric among all automobiles 7 ~ 

in use reach 50 percent? 

3. In what year will the percentage of. households reach 50 percent r 

that are equipped with computer consoles tied to a central com- 

puter and data bank? 

4. By what year will the per-capita amount of personal cash trans- 

actions (in 1967 dollars) be reduced to one tenth of whet it is now? ; 

5. In what year will power generated by thermonuclear fusion be- r““^ — 

come commercially competitive with hydroelectric power? |__ 

6. By what year will it be possible by commercial carriers to get from 

New York's Times Square to San Francisco's Union Square in half L_ 

the time that is now required to make that trip? 

7. In whot year will a man for the first time trovel to the Moon, stay 

for at least one month, and return to Earth? 

* "Never" is also an acceptable answer. 

Please also answer the following question, and give your name (this for identification purposes 
during the exercise only; no opinions will be attributed to a particular person). 

CHECK ONE: 

□ 

• □ 

. □ 

NAME (block letters please): 


I would like 

I am willing but not anxious 
I would prefer not 


to participate in 
the three remaining 
questionnaires 


RANK 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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STEP 2 


From among those participants who indicated on the returned ques- 
tionnaires that they wotdd like to participate in the remaining sessions, Dr. 
Helmer selected twenty-three. The selection attempted to achieve a uniform 
distribution of the number of "e.\|>ert ratings” (i.e., those who ranked 
themselves “1” or ‘‘2,” etc.) for each ot the questions, but was otherwise 
random. Each of the twenty-three participants then received Questionnaire 2 
(see Exhibit 2), containing a summary of the results of Questionnaire 1 (de- 
rived from the responses of the entire group of conference participants). 
Each participant’s Questionnaire #2 also listed the estimates he gave on his 
Questionnaire #1. 


STEP 3 

The completed Questionnaires #2 were returned to Dr. Helmer and 
analyzed, and each participant then received a copy of Questionnaire #3 
(see Exhibit 3). The participant’s Questionnaire #2 was also returned to 
him. 


STEP 4 

The completed Questionnaires #3 w'ere returned to Dr. Helmer. These 
were analyzed and each participant then received a copy of Questionnaire 
#4 (see Exhibit 4). The participant’s Questionnaire #3 was also returned to 
him. 


STEP 5 

The series of questionnaires were analyzed by Dr. Helmer and Mr. T. J. 

Gordon, and the results were presented to the entire group of conference 
participants. The results are summarized in Exhibits 5-8. 

Participants found the session to be extremely educational. It also raised 
a number of questions relating to the importance of proper question formu- 
lation, the selection and ranking of experts, and the tendency to move 
toward the interquartile range to avoid extra effort (i.e., the justification of 
one’s position in writing). Such illustrations were believed to be very valu- 
able for imparting knowledge of the technique, even though the results of 
the forecasts, in this case, were not expected to be particularly reliable. 
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The some familiar seven questions ore restated below, together with the medians and interquartile ranges (IQRs) or the 23 second-round responses. 
Also Included are some brief arguments as to why the estimates should be either earlier or later than those within the IQR. 

Please reconsider your previous estimates (which are attached), and revise them if you wish, giving the stated reasons for raising or lowering them 
what weight you think they deserve. (If there is no change in your previous response, please re-insert it under “Your new answer.") 

If your present answer ties outside the indicated IQR, briefly state in the tost column why you think, the argument that had been given in favor of 
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to an earlier date. sisl this. 
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countored by positive response of electric 
utilities. 

The loter dote proponents seem to over- 
look the possibility of the car to tie Into 



EXHIBIT 5 


SUMMARY OF OUTCOME 



Inter Quartile 
Range 

Median 

"Expert" 

Median 

I- Family income doubled 

1982-90 

1985 

1985 

2. Electric autos 50% 

1985-2000 

1995 

1997 

3. Home computer consoles 

1985-2075 

2010 

1985 

4. Credit card economy 

1 985-90 

1985 

1987)4 

5. Economical fusion power 

1985-2030 

1990 

1987)4 

6. N.Y. *-> S.F. in Vi time 

1 975-80 

1976 

1975 

7. Man on moon one month 

1975-80 

1977 

1975)4 


• Median of the 8 individuals who ranked themselves highest. The cutoff 
point was usually 2 or 3. 


EXHIBIT 6 

CONVERGENCE OF RANGE WITH SUCCESSIVE QUESTIONNAIRES 
(Heavy Lines — Range, Dotted Lines = Final Median) 



uj 2020- 
< 2010- 
q 2 000 

uj 090 - 
y 1980- 
S 1970- 
a 1960- 



QUESTiONNAiRE 


# = \ 


j i l 

2 3 4 



QUESTION 5 


QUESTION 6 


QUESTION 7 
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EXHIBIT 7 



COMPARISON WITH OTHER STUDIES 




(Mediant) 




Present 

Conference 

RAND * 


1963 ?* 


Forecast 

Pretest 


LRF Study 

1. Family income 

1985 

1985 

- 


doubled 





2. Electric 
auto* 50% 

1997 

1988 

. 


3. Home computer 

2010 

2002 

2005 

Facsimile 

consoles 




magazines 
in home 

4. Credit card 

1987 

1995 

1974 

Credit link 

economy 




bank to 
stores 

5. Economical 

1988 

1990 

1986 

Controlled 

fusion power 




fusion 

achieved 

6. N.Y. S.F. 

1975 

1980 




in Yi time 





7. Man on moon 

1975 . 

1980 

1975 

Two men 

one month 




on moon 





one month 

• A 1966 Delphi pretest, using 23 RAND 

employees as participants. 



•• A 1963 RAND Long Range Forecasting Study. A report of this study appeared 
as an appendix to Social Technology, by O. Helmer (Basic Books, 1966). 


EXHIBIT 8 

FORECAST PRECISION 

(Note: Numbert on Graph Refer to Question! Asked) 
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3.3 ADVANTAGES OF THE DELPHI METHOD 

The Delphi method is probably the most progressive technique to be applied to 
technical or scientific forecasting. 

The main advantage of this method is that it does away with the pressures which 
can occur at committee gatherings due to possible personality conflicts by a few or a 
majority of the members. In a Delphi exercise, anonymity prevails and each partic- 
ipant or respondee can freely state his own opinion. 

Another advantage of the Delphi method is that it provides, through the use of 
multiple rounds, a feedback mechanism beneficial to both the respondents and the 
evaluators of the answers. To the former, because they must constantly submit their 
answers to their own reevaluation, and provide justification for answers that fall out- 
side of the range of majority thinking. To the latter, because they may modify the 
scope and depth of the questions to be submitted during one round on the basis of the 
answers obtained during preceding rounds. 

The results of a Delphi exercise may be precisely presented in terms of two 
simple statistics, the median and the interquartile range. Variations of these statis- 
tics from round -to-round offer the evaluators a valuable tool to determine if a con- 
sensus is being obtained. 

The size of the panel need not be as large as would be required in a statistical 
sampling of a population of experts, since the Delphi method does not rely on a Gaus- 
sian distribution of the sampled population to give meaningful results. Moreover, it 
has been found that large groups do not necessarily improve the quality of the answers. 

The Delphi method affords a convenient means to test itself and improve on itself 
by having panel members arrive at a consensus about a question whose answer is 
already known. This is also a good way to test the expertise of a panel on the subject 
on-hand. Such an experimental group opinion situation affords also the means to check 
on the accuracy of the method. 
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Even if a consensus among members is not obtained, the Delphi method serves 
to crystallize the reasoning process that might lead to one or several positions on an 
issue and thus helps to clarify that issue even in the absence of a group consensus. 

3.4 LIMITATIONS IN RESULTS OF DELPHI METHOD 

There is a body of thought prevailing that experts in any one particular field do 
not really hold the key to forecasting wisdom relative to that field and that collected 
errors in their forecast, although presented in a suitable mathematical fashion, are 
still errors. This argument may be countered to an extent by establishing a multi- 
disciplinary panel of experts. In attempting to arrive at a consensus, respondents are 
asked to answer the same questions over several iterations and to defend outlier 
opinions. This can produce pressures on some respondents to "give -in" and "go along 
with the crowd, "thus moving their estimate closer to the median. Although this is 
possible and likely does occur, it may be minimized by careful selection of panel ex- 
perts who will, when they feel they have a good argument for a deviant opinion, refine 
and develop their case. 

The anonymity, which can be an advantage of the Delphi method, may tend to 
reduce communications and eliminate the often stimulating direct verbal interchange 
between persons in committee and brainstorming sessions. Anonymity combined with 
feedback tends to make the whole process of evaluation cumbersome and time- 
consuming. This is why the choice of the size of the panel is such a delicate operation, 
given the time available for the whole survey. 

It is very difficult to eliminate bias completely from the Delphi method: there 
could be bias on the part of the administrator, and the pressure to conform mentioned 
above is also a kind of bias. Relative advantages of the committee and Delphi tech- 
niques are still being debated and proponents of both present valid arguments. 

The Delphi method has also been criticized because it yields a set of "linearly 
independent" estimates of the future, with the probability or date of each item esti- 
mated independently of the others. The answer to this objection is to see the future as 
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a large tree network with a multitude of paths. In this model, the realization of cer- 
tain events will make other alternatives more or less probable. See also cross -impact 
analysis described in Paragraph 5. 8 and Appendix A. 

Once again, the complications inherent in a model that allows for cross- 
correlations must be weighted against the need for obtaining answers from the panel 
within a reasonable amount of time. 

Note that the role of the evaluators is much greater in a Delphi exercise than in 
committee work, where their task is practically completed once the committee has 
been selected and tasked. In a Delphi exercise, the evaluators' task just begins with 
the assembling of the panel. The foregoing limitations reveal some of the hesitancy 
in applying any new method. With practice and the accumulation of further data, more 
questions will be answered about the Delphi method, thus hopefully mitigating most of 
the objections to its use. 
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SECTION 4 - PROCEDURE FOR DELPHI APPLICATION TO 
LONG-TERM NATIONAL TELECOMMUNICATION NEEDS 


4. 1 SELECTION OF COMMUNICATION PANEL MEMBERS. 

The selection of communication panel members is predicated upon the nature of 
national communications, covered in Appendices B and C. Such choice is highly 
dependent on the character and density of communications in relation to the distribu- 
tion of population throughout the country. Finally, panel selection reflects both the 
predictive quality and the highly technical orientation associated with determination 
of telecommunications needs and priorities. 

First, consider communication scientists as panel members who are familiar 
with the work being carried out at the frontiers of knowledge in their respective field: 
they will have to blend the quality of long-range vision with technical competence in 
order to predict the nature of communications in the 1990s. 

Second, communications engineers, well versed in the technical aspects of their 
field, are required on the panel. It is preferred that they be familiar with technology 
forecasting, and have a "down-to-earth, " practical approach broadened by experience 
in economic and social fields where possible. 

Third, the panel should include communication policy experts who have decisive- 
ly contributed to, or who have made major decisions affecting national communica- 
tions (e.g. , persons who have prepared dockets for the FCC). 

The membership of the panel is not limited to communication specialists. Since 
economic, cultural, and societal factors determine the distribution of population, the 
panel should include representation from such fields as needed to form a composite 
picture of population trends and consequent communication needs in the 1990s. 

Changes in values are certain to influence the development of technology in 
general and of communications in particular. All panel members and particularly the 
social scientists are expected to be aware of chan g ing values such as the growing con- 
cern of society with the impact of technological growth. 
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Long-term forecasting has usually been thought of as belonging to the realm of 
the scientist. However, previous exposure to forecasting is desirable for all members 
of the panel because it is a discipline that is consistently called upon during an actual 
Delphi exercise looking to the future. 

In choosing the members of the panel, attention should be paid to their fore- 
casting "temperament" and a proper unit of extrapolators, goal setters, and cyber- 

18 

neticists should be appointed. 

The recommended mix of panel members include representatives drawn from 
universities, Government agencies, private companies, and nonprofit organizations. 

A suggested list of organizations is presented below which is not intended to be all 
inclusive. The procedures for selection, detailed in the following paragraphs, are 
based on a synthesis of national communications as shown in Appendix B, and on the 
considerations expressed above. A large number of experts should be selected in 
order to form a preliminary list extensive enough to permit sufficient choice of a 
multidisciplinary panel adequate for the study. 

The experts comprising the preliminary list must be apprised of the purposes of 
the exercise, also obtain their views on study objectives. The members of the panel 
finally selected must subsequently receive more extensive and more detailed orienta- 
tion to cover the objectives and procedures of the study. 

Step 1 - To ensure that the composition of the panel presents the proper mix and 
is adequate for the prediction of trends in national communications, the following set 
of procedures for obtaining a preliminary list of panel members is recommended: 

1. Contact the Dean of Engineering of the universities listed below and solicit 
the names of at least five communication engineers or scientists to be 
recommended as members of the panel. The universities suggested are: 

California Institute of Technology 
Columbia University 
Massachusetts Institute of Technology 
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Purdue University 
Stanford University 

State University of New York (Polytechnic Institute of New York) 
University of California 
University of Illinois 
University of Michigan 

2. Contact the Managing Director or President of the following organizations 
and also seek from them the names of at least five suitable candidates: 

Air Transport Association (ATA) 

Stanford Research Institute 

The American Federation of Information Processing Societies 

The Armed Forces Communications and Electronics Association (AFCEA) 

The Association for Computing Machinery (ACM) 

The Association of Federal Communication Consulting Engineers (AFCCE) 
The Bell Telephone Laboratories 

The Communications Systems Group of the Institute of Electrical 
and Electronic Engineers 

The Defense Communications Agency (DCA) 

The Department of Commerce's "Environmental Sciences Services 
Administration" (ESSA) 

The Federal Aviation Administration (FAA) 

The Federal Communications Commission (FCC) 

The Institute of the Future 

The Maritime Mobile Community 

The Mitre Corporation 

The National Aeronautical Science Administration (NASA) 

The Office of Telecommunications Policy 
The Rand Corporation 
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3. Contact the Director of Engineering of the following corporations and obtain 
at least five suggestions from them: 

American Satellite Corporation 

ARINC 

COMSAT 

DATRAN 

General Electric 

General Motors 

General Telephone and Electronics 
International Business Machines 
International Telegraph and Telephone 
MCI 

Radio Corporation of America 

The American Telegraph and Telephone Corporation 

Western Union 

Xerox 

The advantage of the method just described is that it is capable of giving a total 

of at least 160 potential panel members. If the number obtained is less than expected 

it means that the name of some of the communications experts were furnished by more 

than one source, which may be considered as an additional endorsement. Admittedly, 

there are problems in selecting experts for any type of long-range forecasting. In 

19 

"Problems of Selecting Experts for Delphi Exercises," Gordon Welty, of the 
American University, mentions that it is difficult to distinguish among levels' of 
expertise, and that the method of self-assessment has been found wanting. The need 
to use experts for the forecasting of technological breakthroughs is not in question 
here. As to the method of choosing a preliminary list, it is simply illustrated by the 
idea that "if a man is considered an expert by what amounts to be an extended peer 
group, then he must be one." As an analogy, a good technical reference is one that 
is consulted a lot. 
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When the previously enumerated sources of potential panel members are con- 
tacted, they should be apprised clearly of study objectives (i. e, , forecasting national 
communication needs), and the types of expertise required (scientist - long range; 
engineer - technical; decisionmaker - legal, economic; social scientist - communi- 
cations, behavior, ethics). The determination of the forecasting "temperament" of 
the candidate experts, who are nominated, should be left for the next round of 
selection. 

It is possible that some of the people contacted will be at a loss to name an 
expert in the social sciences. Should this happen, it is suggested that such persons 
be requested to name someone else who may be able to furnish this information, e. g. , 
the Dean of the Social Sciences Department of a university. 

Step 2 - Next, determine the optimum size for the panel. The question is: 

"how many experts are needed to give a good answer to meet Delphi study objectives?" 
When the specific answers sought are of a quantitative nature, this question may be 
answered by considering how large an error in the estimate can be tolerated and how 
great a risk of exceeding that error should be taken. This approach cannot be used 
if the questions are not well defined and if at least some are not of a quantitative 
nature. The number of panelists can be determined somewhat arbitrarily but must 
be consistently chosen, taking into account the following factors: 

1. The panel may be larger than that of a committee doing the same type of 
work, 

2. It should not be so large as to render the whole Delphi process too 
lengthy. 

3. The panel should be large enough to validate the statistics to be applied 
to the answers. 

4. Given these considerations, a panel size of about 40 members is a 
reasonable objective. It is also a practical one since it represents an 
acceptance rate of about 1/4 of the maximum available number of 
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candidates. Out of the 40 members, at least five should be social- 
scientists. If the scope of the study is sufficiently broad, the panel 
may be expanded to 50-60 members and may be broken down to as many 
as 10 subpanels, each assigned an area of specialization. 

Step 3 - Having obtained a preliminary list of experts from universities, 
government, and industry, the next task is to select from them the members of the 
panel. The approximate 160 candidates obtained from the first contact should be 
given a description of the forecasting exercise and asked if they are willing to par- 
ticipate. Once their willingness to undergo the complete Delphi exercise has been 
obtained, they should be asked to answer some questions the purpose of which is to 
categorize them in any one of the following four forecasting attitudes as described 
in Reference 18. 

Discounters : Interested only in near-term problems, disinterested in 
forecasts. 

Extrapolators : Consider the future as an extension of the past, put emphasis 
on data, use extrapolation. 

Goal Setters : Believe that the future can be created, put emphasis on values, 
use narrative analyses. 

Cyberneticists : Combine the past and future creative approaches, are adap- 
tive, use the interaction of exploratory and narrative forecasting. 

The results of the questionnaire will help to eliminate the discounters from the 
panel and ensure the predominance of extrapolators, goal setters, and cybernet- 
icists, the latter to counterbalance the opposite but necessary tendencies of the two 
other types. 

This latter step takes into account the "forecasting temperament" of the can- 
didates and the number of referrals given. Other factors to consider in final 
selections are a candidate's particular area of expertise and his degree of exposure 
to other areas of knowledge. 
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4.2 GENERATION OF SIGNIFICANT QUESTIONS 

The construction of a questionnaire for a Delphi experiment poses several 
problems for the experiment designer. Each question must be so constructed that 
it reflects a valid possible future condition, yet at the same time it must avoid the 
pitfall of being so worded that it biases the participants' responses. Additionally, 
the questionnaire designer is ordinarily tasked with providing the back-up material 
for the participants' use (or not, depending upon their individual preferences) in 
formulating their responses. This increased responsibility provides further oppor- 
tunity for inadvertent bias, which must be avoided to the greatest extent possible. 

The general method for the creation of a Delphi questionnaire is for the 
experiment designer to seek the assistance, of highly qualified individuals in formu- 
lating the questions to be asked and then to subject those questions to close scrutiny 
by psychologists or other social scientists experienced in questionnaire development. 
The role of the social scientist is to examine the structure of each question and of 
the questionnaire itself for comprehensibility, bias, and likelihood of producing a 
useful set of results. If the experiment designer is able to fill either of these roles 
himself, so much the better, but if he is in the slightest doubt concerning his 
expertise he should demand the professional assistance which will assure the 
success of his experiment. 

In seeking the assistance of the most qualified experts in designing the question- 
naire, one may eliminate such experts from consideration as panel members who 
may be the best source of Delphi data. While this is true, it is of no more con- 
sequence than the same objection raised to the appointment of the most skilled 
expert to occupy the objective chairmanship of a regular committee. His contribu- 
tion in the objective, guiding, analytical role will always outweigh his potential con- 
tribution as a panel member. 

4.2.1 Structured and Unstructured Questionnaires 

Questionnaires fall into two general types: structured and unstructured. 

Where the purpose of the questionnaire is to develop statistically useful data, it is 
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necessary to use a closely structured approach, that is, each question must be such 
that it may be assigned a value. Yes-no, percentage estimate, probability, and other 
rating-scale questions are of this type. Very obviously, from the standpoint of the 
Delphi method, it is necessary that questions be carefully structured. 

And yet, what assurance is there that in the construction of the questionnaire 

the experiment designer has addressed all or even the most significant of the issues 

impinging upon his area of investigation? That the answer to the question is a 

resounding "none" is very well illustrated in the author's own critique of his RAND 

2 

long-range forecasting study. 

"First of all, we would like to register our surprise at some of the ideas 
that have propounded. To other persons, of course, a different set of 
items might be the unexpected ones which we had failed to anticipate: 

The implication that the water- covered portions of the earth 
may become important enough to warrant national territorial 
claims. 


The probability, in the relatively near future, of very wide- 
spread use of personality-control drugs. 

• • • 

The fact that control of gravity was not rejected outright. . . " 

And so on, for a rather lengthy list, considering the qualifications of the individuals 
who prepared the experiment. 

The fairly obvious conclusion is that unstructured, or open-end questions must 
also be included in any Delphi experiment, if for no other reason than to discover 
those areas which have not been properly addressed. There are other reasons why 
and when an unstructured question should be used, and these are addressed sub- 
sequently in this section. For the moment, let it suffice that both structured questions 
and unstructured questions which can lead to discovery are essential ingredients in the 
Delphi mix. 
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4. 2. 2 Order 


It is axiomatic in the construction of any questionnaire that its order appear 
logical to the respondent. It is important to remember that even in a Delphi panel, 
composed of experts, perception of relationships is highly individual, so that the rela- 
tionship the experiment designer wishes to explore may not be an obvious one to the 
respondent. For example, a respondent may see a definite relationship between 
message switching in the range of a microsecond and the development of extremely 
wide bandwidth laser carrier systems. To the researcher this may be an obvious 
relationship but one which is less important to him than (to stretch a point) that 
between fast message switching and the level of industrialization in Latin America. 
Even if it is possible to expand the question to include the reasons for grouping it 
with another, it is still wise to order the questionnaire in such a way that the obvious 
relationships are exposed. Not only does such a construction method aid the respond- 
ent to order his thinking, it reduces the bias and distortion that may result from over- 
emphasizing the experimenter's conception of the cross-impacts which may exist in 
his experiment. 

4.2.3 Quantification 

The general form of the Delphi investigation lends itself readily to quantifica- 
tion, which form might be stated, "What is the probability that event X will happen 

within time frame Y? " Once the time frame is chosen quantification becomes a 

\ 

matter of understanding what the respondents mean by the probabilities they choose. 

i 

This is often not as simple as it might appear. The experimenter has no way 
of looking into the minds of his respondents to ascertain that one man's 20 percent 
is not another man's 45 percent. Using verbal classifications is equally ambiguous: 
what, to a group of respondents, is the variation in meaning among likely, very 
likely, and extremely likely? In the latter example, all one needs to do is to provide 
a null (central) point between the likely side and the unlikely side of the question and 
he has reduced his data effectively to a simple three-value answer whereupon he 
might as well eliminate the rest. 
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There have been many ingenious attempts to derive some sort of confidence in 
the accuracy of the qualitative measures applied to Delphi questions, and in the 
absence of any real means for evaluating their adequacy, it might prove wise for the 
experiment designer to try all of them (as well as to apply what is known about human 
responses). Among the more interesting innovations are these: 

• Instruction in the location of the 50 percent confidence point. Since Delphi 

is a voting system, the respondents are carefully instructed (on the question- 
naire) that a 50 percent probability is that point at which he would be as 
content to wager that an event would occur as that it would not, or that it 
would occur before a date as after. Examples are usually given in terms 
of the actual experiment. 

• Pairing of numerical probabilities with corresponding verbal statements, 
such as *'10 percent = could possibly happen, but so unlikely as to be insig- 
nificant" or "90 percent = Virtually certain, but by remote chance might not 

l 

occur at this point. " 

• Paired choice. The respondent is required to choose the timeframe in 
which some event will absolutely occur and given a companion question in 
which he must choose the frame in which it will absolutely not occur. There 
is little ambiguity with this method, but it also limits the total amount of 
information which can be derived from the question. 

Figure 4-1, from Reference 2, illustrates four rounds of Delphi questionnaires 
and several means of writing questions so that they may be quantified. In the first 
iteration, questionnaire 1, the authors stated the purpose of their investigation and 
requested that the respondents select, individually, those topics which they felt should 
be studied in the Delphi. While this form of question is totally free-form, it can 
result in a list which can be quantified through analysis of the submitted topics, clas- 
sifying those that seem to be the same or nearly so as identical. A choice can then be 
made of the most frequently occurring topics; these can then be used for the second 
iteration of the questionnaire. Alternatively, the first iteration could be a listing of 
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LOHG-IAIGE FORECiSTIKG 8TUDI 


1.1. SCIEJfTLF IC BRElXTHBODGHS 

Ona of tb# Major probltsa of conducting a predictlwa atudy 
which posoa Its questions on tha basis of sztrspolatlonM of 
currant technology is the alaost unavoidable exclusion of dis- 
continuous state-of-the-art advances. 

In this currant study a period of 50 years is being con- 
sidered. It la possible that inventions and discoveries not yet 
visualised could have a Major In pact on our society during this 
interval. It is easy to observe that the pace of scientific and 
technological innovation hss been steadily increasing and that 
the time between origination and application has been decreasing. 
Therefore we beliove that cany generations of Inventions can 
find application during tha period under study. 

Sobs insight even into discontinuous stats-of-tbe-ert 
advances Might perhaps ba gained by exaalnlng tha world's need 
for such advances, in view of the old trulsa that necessity is 
the Mother of invention. Therefore, you are asked to list below 
major inventions and scientific breakthroughs in areas of special 
concern to you which you regard as both urgently needed and 
feasible within the next 50 years: 

Do you know of the sxlstence of any information. In the for* 
of tabulations or analyses, that sight he particularly valuable 
In reaching projections of the kind requested? 


i .2 screw me breaktorougbs 

s. Listed below In Table 1. 2o ere oost of the scientific breakthroughs 
suggested by the respondent* ae potentially possible during the 
next 50 years. Please Indicate your Judge* nt of the probability 
of iBplamatatlom during each period. Sot* that the crushers 
Inserted by you In each row should add up to 100. (In the case 
of l ten* Involving gradual development such as synthetic food 
pr o du ction or autceated. education, "lcplenentatloo* should be 
Interpreted a* referring to the tins free which the effect on our 
society will no longer be negligible.) 

b. Considering the breakthroughs suggested la Table 1.2a are there 
other potential breakthrough* which you would car* to add? Vhen 
do you believe they will occur! Please asks your additions In 

Table 1 . 2 b. 


8UMUS or apmora as scuts up ic tcatthboucss 


1 =? ? f 3 ¥ 

5 $ Pis 


7- efficient Idea-coding to convey 
precise Information independent 
of language 


Physica l __ 

1. Refortmtlon of physical theory, 
ellulnatlng confusion in quar.tusi- 
relatlvdty and simplifying 
rtlele theo 


Klolaturl ration of electronics 
carried to the molecular level 


Figure 4-1. Question Qualification (Page 1 of 3) 
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i.« jciomnc Kurnnais 

This U Iht Ult «unUanln In **r pmnit U*r.U ry into ,»1 fvl n on ictentlfle ««•»- 
tkr««M. 

brort |l>ln you an ittount of wh*t cwurntm lot* to Ian mttt 4 tn th» 

precedlrm round and put tine iw further substantia* questions baron t*j , we would Ilka 
to your prvfvrwr.ct at to »rwtn»r or not a ru port an this atu 3 j anou'.d wentlon your 

participation a* • reipondent. Oxtar no ctrciaaUMtt wilt any tlnct IMUIlora ba 
attributed to you. 

a. exact hor* If you nova no objection to ha*U« your wa own t lined aa that or a mydaBt 
In tMi currant IU1| 1 . — . 


»- Table ].l| an t(* n»»t papa lltta a Mober af potential darala^ wf itt la tha orgnnl- 
tatlanal and operational aethoda of aclanttflc Investigation. Cf the#*, th* n rat 
three ara repeats fro* Suatt lotmalra J; tha others »ra additional Haas p ropoowd by 
th* raapondanta far eona 1 deration. 

Thu* for, there la no elMPOtt consensu# regarding either Cfta probtaillty of occurrence 
ar the desirability of the flr»t thrst U»u. HI ara sonsltered cottar llaaty and. 
on trio average. moderate benefit. Over tra nett 25 ftir*. Itao J la considered 

slightly mjr* llkaly than Itaa 2 . and I tea } slightly aura likely than Item 1. Aa for 
their potential beraflta, th# order It reversed: Item J la thought to ba ellghtly 

oora beneficial than Item 2 , and Itaa 2 slightly tore beneficial ttan Itea 3 . 

In concluding the eent 1 deration of thate potential d#**lop*«ent* In th# organisational 
and a pa rational methods of tclmtlfle mean igatton, aa aexild lit* you to rank tha 
el|ht line In Table l.»b fro* 1 to 8 In two reapectai TTw probable degree of thalr 
reallaatlon attain the nett 23 yea ro, and the detirablllty af tnalr rtallsatlM. 


table l.«b 


H 

Potential deaelopwanti 

In the organic* t lone 1 and operational Methods 
Of scientific Investigation 

hank* 

by expected 
degree Of 
realisation 
within 23 

gaCSBMj 

Sill 

i 

deform af present nodes of scientific corwrunlcatlon through 
the use of autoaated Information retrleeal eystews 



2 

Maori entatlon af trlmtlflc nethooolocy toward greater 
Interalselpllnsr-y cooperation 



3 

widespread use of slaulstlon for experimentation la the 
social sciences 



* 

Increased eapnoil* on basis research la gorarrsesnt-eupported 
D « D 



I 

uiJespresd use In scientific research of more Intelligent 
machine*, with computers on tn* level er *eel leagues* 
rather than sf “eer-rsnt*’ 




Synthetic sttrailatlon af Individual Invent 1 senses (by chsml- 
ctl Incectien, eesvb.-al wanlpuUtlan, external rewards, etoj 



Ih 




’ 

Shift In approach to blmndtcsl problem froo predominant 
rape risen tat lan to nor* theoretical ground 




’»• tank 1 far hi -.heat degree and create it CaalrmblUty . tank • far lowest li p e i and 
least Oaairobl llty. 


or t»a twenty-tea a pacific potential bresktnrounro put before tha panel la tha pcealauo questlon- 
nalre. Item 9 {on r*i\ielrT*, thi probability of *mr through education) and 22 (on r. rowvO-af f»et 
■•chines) ara bain-, dropped from runner consideration (9 because of inn* rant vagueness of th# 
Question. 22 because of eon pa rat lea unlmportano* in tha fbca of a cona 1 dares La divergence af 
aplnlans) . 

On alaeen other It ana a NffUtiiit consensu* hat bean established ta •arrant aur bringing thalr 
canaldcration ta o cloaa, - unless you wish to taka explicit skcaptlon ta any ana af tnaa. 

They ara listed bald" (Tbbi# l.«e) together alth the oojorlty oplnlan an the expects* tlaa af 
thalr occurrence. 


Tbble l.«s 


Item on ahlcn at least a node rat a consensus Mwa ta have bean obtained 




tajorttp opinion 

* 

1 

3HBI 



H 

H 


During the 1970 ’a 


StinilMfl emission ( *Useift 
In the * and Gams cay re- tin* 
of th- spectrum 



1 

Iwnler*e -1 artificial organa ole 
of plastic It) electronic com- 
ponents 



H 

he formation of physical thoory, 

*1 lalnatlng confusion In QimntiM- 
relativity and simplifying particle 
theory 

frobably |i«4Ml deeelepoent .by about 2000 


ID 

Controlled IMiSn mtleir power 

Daring th# period flea l*flO to 2000 


1 


Eatons I «• . though not necessarily universal, 
loemlftl lem during the period frwo about 20 to 
*0 yea ry rmo now 


1 


taring tha period from 1990 to 2930 



Figure 4-1 


tajorlty opinion 


Ascription af br**kthrw*«h 

5 1 Pea* l kt llty af oan-mehlna sym- 
biosis, enabling aan ta sstend m 
lata 1 licence by direct eletlres- 
•aleal Interaction Wlw Ml 
n and * tMgitlig »CM»* 

1 (control af traatty through am* 

I or aodirieatlai of the gruel 
| tatlorwl field 


[taring tha Period fna 1990 to 2030 


jsot for U» year* , 1 


[sot for 200 years. If eeor 


a. P loose aarh etth a croas in tha last cstun of tha a base table any It an* regarding *M rh 
you find youraslf In auaatantlal disagreement with tha stated majority opinion. Pur each 
each ease, elate below the nature af you r t U ig wiasn l and, eery briefly, the raoaon for 
ytar differing apUlm 


Thera are now nine real own L item requiring farther examination. They are listed In 
Tbbla l.«d be loo. T*w table also Includes, ns Item 23 to 23. thro* items about which a 
tents ti ea consensus had already bean announced tn Table 1.3s af tha prealmta Questionnaire : 
thalr consideration la being re i used here because In each c»a* laearal respondents explicitly 
role ad IlngmsMt oltn the a pin Ion stated. 

4. Please '•consider the potential scientific and technological brosktbroush* listed in the 
following table. Thar bra the renal ring ttena on which tnua fsr no esttafectory consensus 
MS been attained. In Some cases, the description has been reworded tn an effort, to 
sllnlnate rag wanes* (which any here been tartly responsible for the apparent discrepancy 
Of aplnlans 1 . 

X» giving im once aor* your ntlMtit af the years when tha taoboblllty or occurrence 
reaches 3 O', and W raspoctlrely , Please take such re ferwu lotions Into secowtt, at well 
as tha aarlts of any atatewwnta af Minority opinions : 


Table l.»d 


E 

lmTbTrnTTTiIM 


Minority opinion 

30S-ye*r 


1 

ssrlly acceptance) of . 
choolcal control over 
sees heredltanr deTecto 
by eediricstlon or geno* 
through nolecular 
engineering 

Gy 2000 

Will tea* longer er occur never, be- 
cause lb would nrcofsltste Inter- 
vention during sobryonic development, 
when the foetus ts Inaccessible, 
fence sMli require prior devetotawnt 
of technlgueo of gabtatlon In vitro 


i 

1 

1 

1 

igmi 

By 1990 

If ltv-eltro generation of proteins 
with all essential ulna acids Is 
Intended, than eueh perfect control 
Of polyasy syhthotls and (refill* 
shMistnr are rvqulrei that aor* ;nan 


! 

! 

1 

iin 

rT^' r TB— W ■ 

Vat for 10 
year* but 
by 2000 




1 

widespread and socially 
widely accepted use of 
nonnarcotic drugs (other 
than alewnall for th* 
purpose of producing 
• pacific changes tn per- 
sonality characteristics 

■ 

ill 


■ 



1 

Econonlc feasibility of 
con-wrclal wonufsetur* af 
■any char leal elements 
fro r subatomic building 
lloeks 

Set within J5 
but within 
CO years 

Mwver. despite tecmteAl r»»» 1 bl 1 • t y , 
because atner aethods will be Creeper, 
suen aa »r«c easing af lwa-rrmj* 
natural sou res* or reprocesstmi frsm 
waste 


1 

j 


Peas lbl llty or Halted 
•esther control. In the 
sense of substantially 
affectlnr, rc.-lanal weather 
at acceptable cast 

Within 23 

year* 

’ 



1 

i» 

Economically useful exploi- 
tation of the ocean throw.h 
fiiclr,. eitn the affect af 
prwtuclihi at least MS of 
tr*s world's food 

Mot within 
50 year*, if 



■ 

nb 

EcMoalcally useful e»- 
ploltstlon af tha ocean 
botcon throenn »intn; 
(ether than erf-ehwew 
oil drllltnr) 

Within 29 
year* 


■ 

i 

18 

Peas lbl llty af using 
druc* to raise tne lee* 1 
of ‘Intel licence J other 
than as dietary supplements 
and not In the tens# wf 
fust temporarily raising 

Met within 
23 but within 

90 ywora 

Mover, because the u»* of molecular 
engineering to rats* th* 13 will 
otmo first; also, the effect of drugs 
1 * limited boctui* they cannot affect 
th* tost* synaptic ceramet! vity of 
the train 



I 


1 

Iter# Chon 90 year*, because an entire 
science af blepsyehlatrr oust b* 
built 



1 

QgQDuHIHHB^H 

Met within 
23 year* but 
aeantMlly 

Mover, because eiyintpo are to* 
delicate to be kept tn laetg eeaa ; 
alto because the demand la (weal lan- 
*bi* 


j 

1 


not within 
29 y»* re bit 
eeentuelly 

totwewm asaoag# iW rtrpMi m«M 
Me anaonaseably long 


t 

1 


Rat within 
29 year* and 

psaelbly never 

Definitely eventually, at Ivast far 
• rtlftcltl regen* rbtton of Uobe 

■ 

■ 


o. Bo yaw with to odd gag wapiti' 3i particular, ta there anything further m wish to My In 
support of pomp flea m O Mp r e eeed tn rear entries In the fsregaing table* 
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topics generated by the experiment designer and submitted to the panel for a consensus 
of those topics which merit further consideration. The latter is possibly the preferred 
method, but it should be accompanied by a free-form response in which the panel 
could suggest other topics. 

In the second iteration, questionnaire 2, in Figure 4-1, the designer has chosen 
a time-line probability of occurrence approach. This is a very useful approach for 
most Delphi subjects, but it should be pointed out that only very careful instructions 
will provide any confidence that the probabilities expressed by the individual respond- 
ents have any relation to each other. 

With sufficient instruction, the panel will probably respond in such a way that 
confidence can be placed in the extremes and the mean, so that derived partial values 
can also be useful. 

The third iteration, questionnaire 3 of Figure 4-1, uses a forced choice two- 
valued answer but allows an unstructured response to a specific aspect of the questions. 
This technique might be called semi-structured and has much of the value of both 
approaches, allowing for quantification at the same time that additional insights are 
sought through the respondents' free choice answers. Note that asking the respondents 
to choose any year for their 50 and 90 percent choice points does not unduly compli- 
cate analysis, since the designer may group the years in any way he chooses. On the 
other hand, the respondents are not made to feel restrained by any arbitrary choice of 
time period. 

In the fourth iteration, questionnaire 4 of Figure 4-1, the authors chose to ask 
for rank ordering by the panel of those items on which a reasonable consensus could 
be expected. The number of items to be ranked here is eight, which is very nearly 
the useful limit of ranking. Because Delphi is an averaging technique, longer rank 
lists may be used, but little confidence should be placed in the limited discrimination 
afforded by adjacent ranks. If a large number of ranks is used, quartile values would 
probably be more useful. In Table 1. 4c, in Figure 4-1, the designer has sought two 
things: verification of the consensus he feels has been reached, and exploration of the 
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extremes of disagreement with the consensus. It is well to add this extra step, not so 
much to validate the statistical consensus as to illuminate the real degree of confidence 
which one may place in the final form of the question itself. 

4.2.4 Phrasing the Question 

Precision of question construction is probably the most important requirement 
placed upon the designer of any questionnaire, and certainly a Delphi experiment is not 
immune to this requirement. 

The general rule in the construction of questions is to avoid descriptive words to 
the maximum extent possible. They are confusing, and they tend to bias the question. 
On the other hand, at every opportunity terms should be defined. A question should be 
precise yet retain the simplicity of a direct statement. Legalistic and jargon-filled 
questions should be avoided. 

A general rule of question construction is that each should be unidimensional. 

A question should address one dominant thought or theme and permit only choices that 
are truly parallel and related. Although one of the more interesting aspects of Delphi 
is the opportunity to explore cross-impacts, it is wise not to do this exploration within 
the framework of a single question. Rank ordering may, obviously, involve nonrelated 
items, but where a choice must be made among question elements, they should always 
be truly related. (Don't ask which is more likely controlled, mass psychological con- 
ditioning or economic solar energy, because the respondent will be led to think that a 
relationship is perceived by the designer; on the other hand, including the two items in 
a rank ordering scheme may be permissable so long as direct relationships are not . 
implied. ) 

4.2.5 Special Freedoms Allowed in Delphi 

The Delphi questionnaire designer has freedoms not generally allowed the 
experimenter involved with other forms of questionnaires. For example, the questions 
need not appear easy, since the panel is made up of interested experts who do not 
require this form of sugar coating. It may one day be shown that the absence of a null 
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in a Delphi question is a drawback, but since the respondent always has the freedom 
of no choice, a better form of the null is available and the forced choice question may 
be used with impunity and no fear at all of distorting the results. It may well be that 
for any given Delphi the pros and cons are of much greater value than any consensus 
could be — so that a question may be provocative or open-ended as the designer 
chooses. Finally, in the Delphi, it is expected that questions will be reworded in 
successive iterations of the questionnaire, so that the designer is free to ask those 
questions he considers most meaningful in early iterations, rather than those which 
he considers most likely to produce reliable information. (Obviously, the questions 
on which the final conclusions are based must be those which are most reliable, but 
they may be arrived at by a circuituous route if desired. ) 

4.2.6 Deriving the Delphi Questions 

As stated earlier, the most useful method of deriving Delphi questions is 
through the active efforts of highly qualified experts. On the other hand, as has been 
observed, the panel itself may generate some questions to be asked, once the 
general theme and the broad questions have been decided upon. The latter method 
has the disadvantage that one round of questions is not statistically productive, and 
further, it is difficult to provide the panel with appropriate source and backup 
material in the absence of definition of the questions themselves. It might be useful 
here to propose a method of construction of Delphi questions which has not been 
explored in any of the literature. That method is the assembling of hierarchies 

f 

related to outcome statements. The experimenter may wish to know the total data 
transmission requirement over a distance of 2000 miles or more (or between 
selected points, or at a given transmission rate, or with some other useful qualifica- 
tion) at a given period in time. He might construct several hierarchical branches 
leading toward his main question: classes of data — business, financial, govern- 
mental, scientific; technological capacity to use the data, in the form of computa- 
tional facilities and new equipment; technological capacity to transmit the data; 
costs of use and transmission; economic advantage enjoyed by data users over non- 
users; and so on. He might well then discover tangential relationships which would 
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seem to him to require investigation: dispersion versus concentration of industries; 
the effect of nationalization of industry or the growth of internationalization; the 
impact of industrialization of underdeveloped countries both upon their own use of 
data and upon the already industrialized countries; population growth and concentra- 
tion; the "saturation limit" in cities which produces a theoretical chaos; changes in 
transportation cost and availability; and hundreds of other possible areas of investi- 
gation. 

The construction of such hierarchies can illuminate the experiment for the 
designer in many valuable ways. Time-based relationships can be discovered where 
none were constructed; the availability not only of technology but of money which can 
reasonably be expected to be devoted to the particular event can be considered; 
a priori technological events can be properly related to each other; and sociological, 
governmental, and scientific events can be seen in preliminary relationship. Where 
weaknesses exist in the designers’ own knowledge, those weaknesses can be high- 
lighted and made the subject of early inquiry in the Delphi. 

4. 2. 7 Bias and Ambiguity 

Every method of construction of Delphi questions opens itself to possible bias. 
Where the panel is allowed to formulate the questions to be used, the sheer weight of 
numbers or the emphasis of conviction can lead to biased questions whose outcome is 
certain before the panel is even consulted. When an expert creates a question, he is 
almost certain to have an answer in mind or at least some feeling as to the answer 
and may bias the question to produce that answer. When hierarchies are constructed 
they constitute a pattern, and these patterns can produce bias. 

The avoidance of bias becomes, in the final analysis, an exercise in honesty 
and vigilance on the part of the designer. The elimination of bias is easier — 
because the panel, if encouraged to do so, will respond explicitly and in great detail 
in criticism of any biased question. Further, the iterative nature of the Delphi can 
provide insight into the success or failure of a question to develop real answers. 

This element is discussed in Paragraph 4. 2. 8. 
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Ambiguity in questions is often even less apparent to the framer of the. question 
than is bias. The standard observations concerning any human communication apply: 
the briefer the question the more likely that it will be understood; all descriptive 
words and qualifiers lead to potential or partial understanding and should be reduced 
to a minimum; multidimensional questions will always cause trouble; definition of 
terms is almost always required; a forced choice should always be accompanied by a 
"no choice" option. 

One of the more interesting methods of creating questions to avoid the particular 
form of ambiguity of estimates that results in trivial or useless answers is to make 
the question itself unsymmetrical. One may use an arbitrary group of percentages, 
such as 10, 50, 75, 90, 100 or one may divide time in unsymmetrical groups, such as 
1976-1980, 1981-1990, 1991-2010, or some such arrangement. (The type of time 
division illustrated here also has the advantage of corresponding to the respondents' 
relative inability to make precise predictions of events in the distant future as com- 
pared with the precision of their near-term forecasts. ) The effect of ambiguity of 
perception or estimate can be eliminated fairly well in this manner, provided that the 
choice of asymmetry is logical and reasonable. 

4. 2. 8 Improving the Questions on Successive Iterations 

As has been noted earlier, the experiment designer of a Delphi experiment has 
an invaluable tool in the successive iterations to improve the questions he is asking. 
The respondents are the best of all possible sources of new questions, improved 
(that is, more meaningful, more clear) questions, and of information concerning 
those questions that should simply be eliminated, for whatever reason. Therefore, 
it is imperative that the Delphi contain a method for communicating directly between 
the experimenter and the respondent. Comments should be sought actively and 
repeated opportunities for comment should be included in the questionnaire. Extreme 
positions should be noted for the respondents' benefit at the next iteration (not merely 
at the end of the study. ) Even the sarcastic — and perhaps even the scatalogical — 
comments may be included at the next iteration without harming the experiment in 
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the least, especially if the designers have responded by altering and improving the 
questions being commented upon. Each comment, and each improved question, will 
bring dividends in the form of further suggestions for improving the questions. 

There are other, internal evidences which should be noted at each iteration of 
the questionnaire in an effort to arrive, finally, at the best, most meaningful set of 
questions. One phenomenon which should be investigated is very early convergence, 
or very early consensus. A question on which everyone agrees from the first is very 
likely to be trivial, at best, and if that is so there is no particular loss. On the other 
hand, the question may well have been answered uniformly by the panel from that 
mass of misinformation, "everyone knows." While it may not be true that what 
everyone knows, no one knows, a question that strays into this area can lead to much 
misleading data. (One would assume, for example, that in the 1920s "everyone" 
would have known that harnessmaking was a poor long-range business, since with the 
popularity and availability of the automobile there would be progressively fewer and 
fewer horses — yet today hamessmaking is a thriving business in the United States. ) 
The convergence may be true and valid, but it should be analyzed. 

Extreme divergence is another type of internal evidence which must be investi- 
gated. Usually if the question is at fault in this case it is because of ambiguity, 
although the area under investigation may simply be one in which there is no con- 
sensus possible. Polarization, particularly at the two extremes, is another signal 
to the designer that he has, at the very least, a very interesting question demanding 
further exploration. He may well have constructed a question which actually has two 
opposite meanings for two groups of respondents. If his analysis informs him that the 
question is not ambiguous, he should single it out for special attention at the next 
iteration and seek to discover the reasons for each respondent’s choice of one or 
other of the polarized positions. He may find it necessary in the final iteration to 
investigate the area with a whole new set of questions. 
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4. 3 STATISTICAL ANALYSIS OF ANSWERS TO FIRST ROUND OF QUESTIONNAIRES 


The answers to the questions given in the first round may either be quantitative, 
or qualitative. Quantitative answers are, for instance, the year in which a given 
event will supposedly take place, or the percentage of circuits that might be needed 
in a given year to satisfy a given communication need. Qualitative answers can be 
rated according to their ascending degree of importance. For instance, the answer 
to "how will a given need be met? " can be "not at all" which is given a rating of 0, or 
"in an excellent m ann er, " given a rating of 10. Answers that fit between these 
extremes can be rated accordingly. For instance, if the requirement is not well met, 
the rating may be 4; if the requirement is w;ell met, the rating could be 6. 

In any event the answers to the first round of questions may always be put on a 
quantitative basis and can thus always be subjected to statistical analysis. The 
statistical information desired is the median and the interquartile range (IQR) of the 
first- round questions. (See Appendix A. 2. ) 

The median is the value such that half the observations are below and half above 
this value. The IQR is the range of values which were observed more than 25 percent 
of the time and less than 75 percent of the time. 

Answers that lie in the IQR are chosen as representative of the disagreement 
within the group consensus because these answers lie on each side of the median and 
encompass half the answers given. It is a logical step in the Delphi exercise to 
choose the IQR as a measure of information since the objective is to arrive at an 
answer around which there is the least amount of disagreement. 

Note that the use of the median and of the IQR are inherent in a procedure that 
seeks a majority decision. 

In order to calculate both the median and the IQR a histogram of the values un- 
der consideration is plotted. A histogram is a graph whose abscissa are the values 
collected as answers to the questions and whose ordinate consists of the frequency of 
occurrence of these values. Once a histogram is obtained, the cumulative frequency 
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distribution of the answers can be plotted. Corresponding to a given ordinate we find 

an abscissa which gives the limit below which we have that fraction of the observations 

indicated by the ordinate. This abscissa is called the fractile corresponding to the 

given fraction. The 50 percent fractile is the median, and the 25 percent and 75 

percent fractiles are called quartiles. Thus, the cumulative frequency polygon gives 

20 

the values needed in this analysis. 

A question arises as to the necessity of weighting each answer according to the 
degree of expertise of each participant. If we choose not to weigh the answers the 
randomness of the answers is preserved, i.e. , skewness is not introduced the way it 
might be if weighting were used. Moreover, this does not stop us from taking into 
account the degree of expertise of the participants at a later stage. 

It has been established that the median and the IQR can be obtained in the first 
round of a Delphi exercise. The results may be looked at collectively and some con- 
clusions derived from a comparison of actual results from expected results. For 
instance, it is expected, on the basis of past studies, that as the median date of a 
forecast event extends more and more into the future, the IQR about that date will 
increase, thus reflecting the increasing uncertainty of the panel. For answers that 
do not verify this, it is suggested that the next set of questions probe further around 
the question which produced the "deviant" answer. This is just one illustration of the 
importance of using the evaluation of the statistics obtained to produce a more 
meaningful set of following questions. This feedback mechanism is an important 
feature of the Delphi technique. (See Paragraph 4. 8. ) 

Another statistic that will help in evaluating the answers obtained, is the stand- 
ard deviation, defined as the positive value of the square root of the variance. The 
variance is the average of the square of the deviation of each answer from the mean 
(the mean or average of the observations is in turn defined as the sum of the observed 
values divided by their number). The standard deviation is useful in determining the 

proportion of the population that includes a given range about the mean, given that the 
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distribution is Gaussian. -A comparison of the standard distribution and the IQR 
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will give the evaluator an idea of how close to the normal distribution is the one 
produced by a particular set of answers. 

There will be answers that lie outside the IQR. These answers will not be 
dismissed since they constitute two important elements of the forecast. An answer 
which lies in the lower quartile will denote optimism or pessimism (depending on the 
nature of the question) with respect to answers lying in the IQR, i. e. , closer to the 
median. Similarly, an answer that lies in the upper quartile denotes the opposite 
attitude from that in the lower quartile. Both types of extreme answers will have to 
be validated, in further rounds of questions. In this first round, they give the evalu- 
ator a measure of the extreme positions being taken within the panel and of the corre- 
sponding difficulty in arriving at a majority decision. 

4. 4 FEEDBACK OF RESULTS OF FIRST ROUND ANSWERS TO PANEL AND 
REQUEST TO FILL OUT SECOND ROUND QUESTIONNAIRES 

One of the most important aspects of the Delphi technique is its feedback 
mechanism, whereby the respondents are asked to reevaluate their answers in the 
light of the results and comments obtained in previous rounds. 

In its simplest form, the respondents are given the statistical results of the 
first round of questions (the mean and the IQR) and are asked to reconsider their 
previous answer. They may give the same answer or change it. If their new answer 
falls outside of the IQR (i. e. , if it is an answer given by less than 25 percent or more 
than 75 percent of the respondents), they must state their reason for giving this 
answer. 

Note that there is a veiled attempt, in this second round of questions, to bring 
the answer of each respondent within the IQR. However, this influence of the IQR on 
the respondent is much less than that of a forceful personality in the usual committee 
situation. 

Note also that an answer in the upper quartile is not necessarily considered 
desirable, although it represents an upper limit to more than 75 percent of the 
respondents. This is true because it is also a lower limit to less than 25 percent 
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of the respondents. By means of the Delphi method we want to arrive at an answer 
that reflects as much as possible both a majority opinion and a "centrist" one (as 
reflected by the median). The onus is therefore on the "deviant" respondent to justify 
his answer. This has had the effect, in past studies, of causing those without strong 
convictions to move their estimates closer to the median, while those who felt they 
had a good argument for a "deviant" opinion tended to retain their original estimates 
and defend them. 

Although the expertise of each panel member is not necessarily taken directly 
into account, it is helpful to know their ranking in expertise for each question being 
asked. Since the members of the panel may all be unknown to each other, the method 
the evaluator can use to ascertain the degree of expertise of the participants is to ask 
the latter to evaluate themselves according to a preestablished scale. 

The rating scale may be based on the following evaluation: 


DEGREE OF EXPERTISE 

RATING 

Excellent 

10 

Very Good 

8 

Good 

6 

Not Good 

4 

Bad 

2 

Nil 

0 


This type of evaluation will be useful if weighting of the answers according to exper- 
tise is needed at a later date. 

The Delphi method is flexible enough to allow for a reevaluation of the questions 
being asked. Due to the nature of the subject matter, i. e. , national communications 
needs, it might even be helpful to increase the scope and depth of each question at 
each round of questioning. This may be done after consultation among the evaluators 
and discussion with other communication experts about the results obtained so far 
(see Paragraph 4. 8). 
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4. 5 STATISTICAL ANALYSIS OF ANSWERS TO SECOND ROUND OF 
QUESTIONNAIRES 

The results of the second round of questionnaires has the same statistical 
results as the first round (mean, IQR) plus some brief arguments as to why the esti- 
mates should be either earlier or later than those within the IQR. The selection and 
redaction of these arguments are done by the evaluators. 

A comparison between the mean and IQR after the first round of questions and 
the mean and IQR after the second round of questions will show the evaluators 
whether or not a shift has occurred in the opinion of the group. The nature of this 
shift (positive or negative) may be used to plan the next round of questions. The 
questions may also be influenced by the nature of the arguments obtained to justify 
answers outside the IQR. The changes in response can also be analyzed by the 
following methods: 

1. Calculation of the percentage change in the variance, the standard deviation, 
or the IQR 

2. Calculation of the change in the median 

3. Calculation of the deviation of the results from a known probability distri- 
bution function, such as the Gaussian one. 

The main point is to consider the provision of questions at each round as a process 
that can be made dependent on the results obtained during the previous rounds (see 
Paragraph 4. 8). 

The task of the evaluator becomes more important in the second round of 
questions. He must sort out the arguments that were advanced for extreme positions 
and present them in an orderly and concise fashion so that they may be examined by 
the members of the panel during the next round. 

He must evaluate the shift in opinions that is taking place in the panel and 
determine the extent to which the questions must be changed in the next round to 
accommodate this shift. 
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Some of the evaluators, assessing the results of the second round of questions, 
will either themselves be communication experts or have available the advice of 
communication experts who are themselves not members of the panel. 

4. 6 FEEDBACK OF RESULTS OF SECOND ROUND ANSWERS TO PANEL AND 
REQUEST TO FILL OUT THIRD ROUND QUESTIONNAIRES 

The same familiar questions are restated again, together with the medians and 
the IQR. Also included are some brief arguments as to why the estimates should lie 
outside the range defined by the IQR. 

The members of the panel are asked to reconsider their previous estimates 
(which are attached) and to revise them if they wish. If they revise their answers they 
are asked to give a weight to the stated reasons that apply to their latest decision. 

For instance, if they increase the value of their estimate, they must weigh each one 
of the reasons that are given for increasing the estimate. A respondent whose 
answer still remained outside the IQR is required to state why he was not persuaded 
by the opposing arguments. 

If new questions or modifications of old ones are given to the respondents, they 
are asked to state the reasons for their answer. These reasons will be incorporated 
into the arguments that have already appeared as a result of round two. 

4. 7 STATISTICAL ANALYSIS AND PRESENTATION OF TOTAL RESULTS OF 
QUESTIONNAIRES 

There is no set limit to the number of feedback cycles that occur before the 
final results are tabulated. The actual limit depends on the amount of time available 
to the evaluators and to the members of the panel to come up with the answers, to the 
forecasting problem. 

Following the third round of questions, a fourth one may be envisioned, where 
counterarguments are presented in defense of an earlier figure or of a later figure 
than those limiting the IQR. And again, a revision of the estimate is sought. 
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There will come a point, however, when some or all of the following effects 
will be noticeable: 

1. The spread of opinions around the median will not shrink considerably 
from round to round. 

2. All questions pertaining to a given topic will have been duly explored in 
scope and in depth as a result of the responses of the panel during 
preceding rounds. 

3. The desirability, cost, and feasibility of all occurrences under investi- 
gation will have been duly explored. 

4. The likelihood of all events under consideration will have been suffi- 
ciently explored, i. e. , all possibilities from the "barely possible" to 
the "virtually certain" will have been considered. 

Having all the data on hand, the evaluators enter into the period of data proces- 
sing. Before organizing the data into meaningful results, it is good to remember the 
purpose of the Delphi exercise: 

1. To reach decisions concerning the probable outcome of future events. 

2. To reach those decisions by considering a consensus of the opinions of 
a body of experts. 

3. To present the data obtained from the survey in a clear and concise form 
so as to facilitate the reaching of decisions. 

4. To present enough data so as to give a multidimensional picture of the 
events which have been forecasted. This means that the importance, 
confidence, desirability, cost, feasibility, and likelihood of the event 
will have been touched upon, at least during one round of questioning. 

The median and IQR will thus be clearly presented for each of the questions that were 
raised. Recall that all questions, whether of a quantitative or of a qualitative nature, 
may be rated quantitatively by using some relevant transformation (see Paragraph 4. 3). 
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In presenting the data, it will be interesting to trace the evolution of opinions of 
the panel as a whole and this will be the most noticeable in the way the IQR varies 
from round to round. 

This can be done by using the graphical method shown in Exhibit 6 of Reference 
17, "The Delphi Method - An Illustration" included in Section 3. 


To verify that the spread around the median increases when the median data of a 
an event is pushed further away in time, the format shown in Figure 4-2 from 
Reference 21 is suggested. 



NOTE: A— Flexible internal storage; i.e., easily increased or decreased in size and at will with use 
of plug-in units. 

B— Majority of software built into the hardware; i.e., small packages of integrated circuits 
to be attached to the computer. 

C— Briefcase computers "advanced slide rules" with large memories). 

D— Oral input to the computer. 

E— Laser memory. 

F— One-million-byte memory small enough to be included in an independent desk computer. 
G— Pocket-size computers (^advanced slide rules" with large memories). 

Figure 4-2. Some Results of a Delphi Forecast of Computer 
Improvements and Applications 
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This format is also a clear way of presenting pertinent data. 

Data presented in tabular form allow important questions to be brought out by 

13 

phrasing them as headings of columns. 

Given that the precise questions relative to national communications needs and 
priorities are not yet formulated, an exact statement of how best to present final 
results cannot be made. It can be stated that the best way to present the final data is 
to attempt to maximize its utility to the decisionmaker. Most likely some of the tech- 
niques of presentation discussed above will prove useful. 

It is at this final stage in organizing results that the ranking of panel members 
according to expertise may assist in evaluating the data. The results from "expert" 
subgroups may be obtained and in turn compared with the results from the group 
taken as a whole. 

4. 8 AN EXAMPLE OF THE EVOLUTION OF A QUESTION 

The dynamic aspects of the questionnaire have been stressed and the manner in 
which the results of each round can be used to prepare the questions of the next. A 
simple example will illustrate this. 

Results 

Round Question Median IQR 

1 (1) In what year do you expect widespread 1984 1979-2004 

use of satellites in land mobile 
communications ? 

Interpretation of results: There is a greater tendency to predict a far-distant date 
than one in the relatively near future. Besides repeating the question, the evaluator 
may ask additional exploratory questions, as follows: 
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Round 


Questions 


Results 


2 


(1) As above 

(2) Do you expect that in 10 years Private vehicles: 30% of answers 

land transportation will consist Mass transit: 70% of answers 

mainly of private vehicles or of 

mass transit? 


(3) What is the main technical diffi- 
culty that must be overcome in 
order for satellite communica- 
tions to be applied to land 
mobile vehicles? 


Limit in frequency spectrum: 
18 answers 

Radio frequency terrestrial 
interference: 42 answers 

Lack of downlink power: 

10 answers 


Round Questions 


Results 


3 (1) As above 

(2) As above 

(3) As above 

(4) How do you see the problem of 
radio frequency terrestrial 
interference being solved? 


Restricting use of vehicles: 

> 

10 answers 

Restricting use of communications 
on vehicles: 15 answers 

Reallocation of frequencies: 

15 answers 

Improvements in design of 
antennas and receivers: 

17 answers 


Although not shown here, the answers would be given statistical meaning by 
means already described. Moreover, each round would continue to seek the types of 
explanatory comments typical of the Delphi method. What this example illustrates is 
that the evaluator is free to pursue any avenue of inquiry suggested to him by an inter- 
pretation of the answers obtained in previous rounds. 
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SECTION 5 - DEVELOPMENT OF DELPHI MODIFICATIONS 


5.1 INTRODUCTION 

Although the fundamental concept of the Delphi method is clear, almost every as- 
pect of it is subject to considerable differences in technique. The literature on appli- 
cations of the Delphi technique reveals that almost every study was made with different 
methods used by the panel participants to rate themselves, different uses of this self- 
< rating information, different numbers of topics to be discussed, different numbers and 
types of questions about each topic, different numbers of people on the panel, different 
amounts of knowledge given to the panel members about the identities and backgrounds 
of the other panelists, etc. Sometimes cross-impact analysis is used, and sometimes 
it isn't. When it is used there have been many differences in format, and even more 
important differences in the mathematical treatment of the information collected. The 
physical arrangements for gathering the information vary widely, too, ranging from a 
variety of computerized Delphi conferences run by teletype to printed questionnaires 
processed with varying time delays. Each of these factors can have a significant im- 
pact on the validity of the results and on the time required from the participants. 
Studies have been made of the effects of some of these factors, such as the size of the 
panel or the self-rating of the panelists, on the accuracy of the results. Other factors, 
such as the various techniques of cross-impact analysis, are still being varied on a 
judgmental basis, with no clear agreement on which is the best approach. The para- 
graphs below explore some of these areas, pointing out what is known and what is not 
known about various improvements to the basic Delphi procedure and giving what is be- 
lieved to be the best guidance possible with currently available knowledge. They also 
include some related topics such as cross-support analysis and risk analysis which are 
useful in connecting the results of a Delphi study to a decisionmaking situation. 

5.2 SELF-EVALUATION MATRICES 

Some participants in a Delphi conference will be more expert at answering any 
particular question than other participants are. Those who possess such expertise are 
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usually aware of it. Consequently, it is reasonable to expect that it would be useful to 

ask the participants in a Delphi conference to rate themselves on their expertise with 

respect to each question, and to pay particular attention to the views of those who rate 

22 

themselves highly. Experiment has confirmed this expectation. 

There are several ways in which a person can be asked to rate his own expertise. 
One way is in comparison with the other participants at the conference. If each par- 
ticipant is given some knowledge about the backgrounds of the other participants, he 
can estimate whether he has a relatively large or small knowledge about any particular 
question, compared with the other participants. In such a rating approach, it would be 
possible (and might be accurate) for one participant to rate himself 1 on expertise on 
all questions, and for another to rate himself 5, if a scale from 1 to 5 is being used. 

Another possibility is to ask each participant to note which question he feels most 
expert in, which he feels least expert in, and to arrange all the others in relative rank 
of expertise. In other words, if there are N questions, and a rank of 1 is considered 
least expert, each person would rank himself with a 1 on some question, a 2 on some 
other question, and so on up to N. Only the answers of people who rated themselves 
from, say N-2 to N might be considered in subsequent consideration of the problem. 

One objection to this method is that some individuals might be relatively familiar with 
most of the problems, whereas others might be familiar with only one or two questions. 
If such is the case, it would seem that the participant should be able to indicate it, and 
have the information used. Nevertheless, this system has been tested and found to be 
more accurate than using answers with no self-rating mechanism at all. 

A third possibility is to use the system described in Reference 22. In that study, 
each participant was asked to put down a "5" next to the question he knew most about 
and a "1" next to the question he knew least about. He could then put numbers next to 
the other questions, indicating his relative degree of knowledge about them. He might 
have several 5's, or just one. This, too, has objections, since one participant may not 
know much about any of the questions, and another may know a lot about all of them. 
Even if they are aware of this, this mechanism does not permit them to reveal it, and 
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their answers would be weighted equally as a result. Nevertheless, this method, too, 
was found to give more accurate results than a Delphi conference without any self- 
rating. However, if the participants of the Delphi conference are chosen carefully,, 
there should not be any who don't know much about any of the questions, and the princi- 
pal objection to the last method would be removed. 

Even within the third method there exists a range of interpretation on how to in- 
struct the participants. A 5 could mean that the participant considers himself rela- 
tively expert in the field covered by the question, or it might mean that he has high 
confidence in his answer. Even a person who is an expert in a field will have much 
more confidence in his answers to some questions than to others in that field. It is 
probably better to tell the participants to use the interpretation of confidence level, 
rather than expertise. Studies have shown that if the average confidence rating of a 
group is ? or better on a scale of 5 then the group answer is likely to be close to the 

true answer. If the group confidence level is low, the answer is usually quite differ- 

14 

ent from the true answer. 

Not only are there substantial differences in the way self-rating information is 
defined and collected, there are also substantial differences in the way it is used. 

Some of the possibilities are listed below: 

1. If numerical estimates are being collected, the estimate of each participant 
can be weighted according to his evaluation of his expertise, when computing 
an average answer. This procedure is apparently not common, especially 
since a median answer, rather than an average answer, is usually presented. 
A median is considered better than an average because responses are highly 
skewed. For example, if the question is, "In what year will picturephone 
service come into widespread use?", a single answer of "never" would cause 
the average answer to be "never, " but would have only a small effect on the 
median. 

2. In Delphi rounds after the first, the responses of all participants can be 
tabulated, and the interquartile range and median displayed, but participants 
can be instructed to answer the question only if they rated themselves in the 
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top two or three ranks of expertise. On subsequent rounds only the answers 
of such people would be tabulated and processed. However, participants who 
feel that some of the arguments raised by the experts are faulty would be 
permitted to submit counterarguments. 

3. All participants can be invited to continue to participate in answering ques- 
tions, and the results both for the group as a whole and for the subgroup 
considering themselves relatively expert can be displayed along with the 
interquartile ranges and medians. The number of people in the expert group 
should also be displayed. At the end of the exercise, the results for the 
group as a whole, and for the expert group, can both be published, thus leav- 
ing up to the decisionmakers the problem of which group's estimates to use. 

The second method outlined above seems to be the one generally used. One im- 
portant factor which must be considered in deciding whether to restrict participation 
only to experts is that such action will reduce the size of the group participating in the 
forecast. It has been shown that the accuracy of the results produced by a Delphi group 
depends rather strongly on the size of the group, particularly if the size of the group 
is less than 7. Figure 5-1, obtained from Reference 22, shows the effect of group size 
on accuracy observed in one study. Therefore, if it is noted that the group would be- 
come relatively small if only the experts participate, it might be better to let everyone 
participate, or at least to relax the definition of who is considered an expert. Much 
more experimentation needs to be done before more precise guidance can be given on 
the optimum procedure to maximize expected accuracy, considering the effects of both 
group size and self -assessed expertise. The results of such experiments would prob- 
ably depend heavily upon how much the nonexperts know about a subject as compared 
with the experts. For some questions the guesses of nonexperts might be quite wild, 
whereas for others the nonexperts might do almost as well as the experts. 

5.3 TECHNOLOGICAL ASSESSMENT 

Technology assessment has been defined in various ways. According to Cetron 
23 

and Connor, "Technology assessment is an attempt to establish an early warning 
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system to detect, control, and direct technological changes and developments so as to 

maximize the public good while minimizing the public risks. It is a relatively new 

approach to allocating scientific resources, establishing technological priorities, and 

24 

seeking relevant alternatives to current technology. " Overly says that "technology 
assessment programs are, for the most part, efforts to understand what significant 
impacts proposed and ongoing technological or applied science programs have upon the 
social, economic, and cultural systems. " He goes on to list various societal indica- 
tors, and states, "Each of these societal indicators represents a specific set of in- 
terests - people, organizations, social and economic classes, and special interest 
groups - who, it is assumed either benefit or suffer from a new program. An estimate 

of what actually will happen is the purpose of the technology assessment process. " 

25 

Green says "The basic problem in technology assessment is to assure an adequate 
articulation of potential costs and risks. It is axiomatic, I believe, that in any tech- 
nology assessment the benefits are usually obvious and relatively immediate, and will 
always be more than adequately articulated and pressed by those with vested interests 
in the technology. Costs and risks are, on the other hand, typically much more remote 
and speculative and there rarely are authoritative interests which can be relied upon to 

articulate and press the costs and risks on the assessors and decisionmakers. " 

26 

Gordon and Becker say "Technology assessment involves anticipating the likely im- 
pact of a technology on a variety of hard-to-quantify environmental, social, political, 
and value factors, as well as the effect of potential changes in these areas on the 
acceptability of the technology. . . At its root, the process of technology assessment is 
one of anticipating primary and secondary interactions. The most important of these 

interactions cross disciplinary lines and are therefore difficult to handle analytically." 
27 

Finally, Cetron quotes Gabor Strassers's definition: "Technology assessment is 
nothing more than a systematic planning and forecasting process, delineating options 
and costs, encompassing economic as well as environmental and social considerations, 
that are both external and internal, with special focus on technology -related 'bad' as 
well as 'good' effects." 
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Although all of these definitions differ somewhat, they all seem to be describing 
the same central idea. There is less agreement on exactly how technology assessment 
is done. Most workers in the field agree that it involves use of Delphi techniques, 
cross-support analysis, cross-impact analysis, relevance matrices, and trend extra- 
polation. Overly fits these techniques into a defineable approach as follows: 

1. Establish organization goals and priorities. If a government agency is doing 
the analysis, these would be national goals and departmental goals. Cross- 
support analysis can be used to help establish priorities, since if it is found 
that one goal supports other goals , that goal should get a higher priority than 
if it does not affect them or hinders them. 


2. Predict future developments. This involves trend extrapolation combined 
with subjective judgment of experts, preferably elicited via the Delphi tech- 
nique. KSIM simulation studies and cross-impact studies (described sub- 
sequently in Paragraphs 5.6 and 5.8) are also helpful here. 

3. Link organizational goals and priorities to predicted future developments. 

4. Analyze organization resources. These resources include manpower, 
money, and information. By using these resources which the organization 
has, or can obtain in the future, the organization can influence the future 
course of events. 

5. Develop resource allocation strategies. 

Overly goes on to state that even if these five steps are performed, technology 
assessment may not achieve the objective of assuring that future technological advances 
will have wide-ranging benefits and few detrimental impacts. The major weakness, he 
points out, is that technology assessment, as practiced today, is primarily technology 
oriented, and does not make sufficient use of societal indicators. Adverse effects on the 
quality of life are not always recognized properly. He deals with what can be done, 
rather than with what should be done. Also, he states that future technological develop- 
ments are often too sponsor -oriented. 
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Ralph gives a detailed procedure for doing technology assessment, utilizing 
cross -impact and cross-support analysis. 

In performing technology assessment, it is important to consider the higher- 
order effects of new technology, as well as the first-order effects. While the higher- 
order effects are admittedly difficult to predict, they are sometimes of great 
importance. For example, if every store had access to a packet-switched data net- 
work, an upgraded credit-card-like system in which sales were instantly debited to the 
buyer's account and credited to the store's account might become popular. This would 
cause large changes in the pattern of communications traffic, and might have major 
effects in other fields. For instance, law enforcement might be easier because almost 
everyone would probably have a sort of credit card with his photograph and fingerprints 
on it. This would facilitate identification if the police were given a copy. Forgery, 
which is the crime causing the largest financial loss each year in the U. S. , might be- 
come harder (or easier). Mugging and robbery might decrease, because people 
wouldn't carry much money. (The ED cards would presumably be unusable by anyone 
but the owner. ) The probability or expected extent of such high-order effects can be 
estimated by use of cross-impact analysis or techniques such as KSIM. Before the 
probability of these effects can be estimated, however, the idea that a particular effect 
is worthy of consideration must occur to someone. Use of Delphi conferences with 
fairly large numbers of participants who have good creative imaginations will increase 
the likelihood that the various side effects of an emerging technology will be imagined 
and added to the list of effects to consider. 

One of the problems in technology assessment is that the social, cultural, and 
environmental effects of new technology is difficult to quantify on a common scale of 
value. Nevertheless, it is possible to do some quantification of these effects by rank- 
ing various alternatives. For example, a city with 7 TV stations is preferable to a city 
with only 3 TV stations because of the added variety of entertainment available to the 
residents. With some effort, even such choices can be put on a common dollars-and- 
cents value system, if necessary in the analysis. People can be polled and asked how 
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many cents per month they would pay for the addition of a new TV station to their city. 
With such numeric data one can obtain a numeric value for the desirability index of a 
new project or development, where desirability index is defined as utility multiplied 
by feasibility and divided by cost. 

If the factors can only be ranked, rather than numerically evaluated, it may still 
be possible to eliminate some alternatives on the basis of a low desirability index. 

This can occur if one alternative is less desirable than another in at least one of the 
three variables (utility, feasibility, cost), and no more desirable on either of the other 
two. 

5.4 RISK ANALYSIS 

Risk analysis is an orderly procedure a decisionmaker can use in selecting one 
alternative among many possible courses of action, in a situation where he is making 
a decision under risk. A decision under risk is one in which the utility of the decision 
will be affected by some factors which are not under the decisionmaker's control, but 
whose probability of occurrence can be estimated. 

The general situation is as illustrated in Figure 5-2. The decisionmaker must 

choose one alternative from the set a , a , . . . , a., ... For example, the alter- 

12 j 

natives might be combinations of funding levels for various development programs. 
There are various mutually exclusive alternative possible states of the present and 
future world, s , s ..., s having probabilities p p ..., p ... which affect 

X u K X Ci K 

the results of whatever action he takes. Examples of such states of the future world 
might be combinations of demand for various types of services, breakthroughs in re- 
lated technology, etc. Examples of probabilistic features of the present world which 
may affect the results of his action are the values of certain natural parameters which 
are not known accurately in the geographic area he is interested in, the levels of man- 
made interference of various types, and various other economic, social, political and 
physical factors he may not have firm data on. For each combination of decision a. 

and world state s, . there will be some outcome Y.. shown in the cell of the matrix, 
k’ jk 

The outcome may have many attributes. For example, if a decision is made to fund 
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a- = AN ALTERNATIVE ACTION (A CANDIDATE SOLUTION 
TO THE RECOGNIZED PROBLEM) 

s k = A POSSIBLE STATE OF NATURE 

p k = PROBABILITY OF s k , WHERE I p k = 1 

k 

Y jk = THE OUTCOME ASSOCIATED WITH a - IF s k OCCURS 


Figure 5-2. Decision Elements 



development of a particular communication satellite system, there will be attributes in 
the form of channel capacity, S/N ratio, geographic coverage, mean time before fail- 
ure, etc. Each of these attributes will usually not be known precisely in advance, but 
will itself have a probability distribution, even if the state of the world were known. 

(In this example, the state of the world might have to do with booster reliability, pro- 
jected traffic, and so on. ) In order to be able to make a rational decision, one must 
be able to combine into a single measure of utility the particular values of the attri- 
butes which may occur given that a particular alternative is selected and a particular 
state of the world is found to occur. This single measure of utility can be defined by 
weighting the relative importance of the various attributes, and for each attribute 
creating a normalized curve showing how the utility of that particular attribute varies 
with the magnitude of that attribute. For example, in a communication system the S/N 
ratio might be judged to contribute 15 percent toward the total utility, the remaining 
85 percent being divided between attributes such as geographic coverage, long lifetime, 
large bandwidth, low doppler rates, etc. Within the 15 percent for S/N ratio, it might 
be judged that a S/N of 15 dB or less would have a value of zero, and that utility could 
be considered to increase linearly with S/N ratio (measured in dB) and independently of 
other attribute values, until a S/N ratio of 30 dB is reached, which might be the most 
that could be hoped for with the system being considered. A S/N ratio of 30 dB would 
then contribute 15 utility units toward a total possible utility of 100 units. 

In each cell of the matrix, then, there is a probability distribution for the utility 
of that combination of decision and world state. The probability distribution can be 
obtained from the joint probability distribution of the various attributes in that cell, 
evaluated for utility at each combination of attribute values. The mechanics of obtain- 
ing the probability distribution of utility, and in particular its mean value, can probably 
be done most conveniently by Monte Carlo simulation, in cases where there are several 
attributes which affect each other's utility. Random numbers would be picked and 
plugged into the probability distributions of the individual attributes to find a random 
combination of attribute values, the utility of that combination would be calculated, and 
the process repeated several hundred or several thousand times to find an average 
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value of utility for that cell. If the utility associated with different values of each of 
the attributes is independent of the values of the other attributes, the process becomes 
much simpler. All that one has to do then is to find the mean utility associated with 
each attribute, and add up all of these means, since the expected value of a sum is 
equal to the sum of the expected values. However, it should be noted that the expected 
value of the utility of a given attribute cannot usually be obtained by simply taking the 
expected value of that attribute and computing its utility. The only time such a proced- 
ure works is if the utility of the attribute is linearly proportional to the magnitude of 
the attribute. If the relationship is not linear, the probability distribution of the attri- 
bute must first be transformed to the probability distribution of the utility of the attri- 
bute before the expected value is calculated. 


The entries in each cell of the matrix shown in Figure 5-2 should be the expected 

utility, U , corresponding to that cell. Since in decisionmaking under risk it is 
■ j“ 

assumed that the probability p of each state of the world (each column) can be esti- 
mated, the expected value of each alternative course of action (row) can be calculated 
by simply summing the products of the probabilities in each column multiplied by the 
expected utility of the payoff in that row and column. The recommended decision is, 
of course, the one with the highest expected utility. 


For the above procedure to work, the states of the world listed in the column 
headings should be mutually exclusive and exhaustive, so that their probabilities add 
up to one. If one wishes to allow for some unknown events, one can label one of the 
columns "other, " and give it a probability. 

Another way of visualizing the decisionmaking process and the risks and payoffs 
is a decision tree, such as the one shown in Figure 5-3. The decision tree can show 
the same information presented in Figure 5-2, and it can also handle multiple -stage 
decisions which are difficult to handle using the matrix approach. 


The initial branches of the tree represent the set of alternatives a.; they stem 
from the initial decision to be made by the decisionmaker from a decision fork. A 
decision fork is represented by a square. 
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dj = CANDIDATE ALTERNATIVE 

p k = PROBABILITY OF STATE OF NATURE 

u lk = ? u ( Yj ) k f (Vj>i k ciVj 


Figure 5-3. Decision Tree Representation of 
Decion Model 
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Each branch representing an alternative a. terminates in a fork with branches 
representing the states of the world. The branch to be taken at this fork is determined 
by chance; a chance fork is represented by a circle. A probability p^ is associated 
with each of these branches. These branches can lead to outcomes or to other decision 
or probability forks. 


Each path from initial decision fork through an outcome branch yields a payoff, an 

expected utility u , which is obtained by combining utility functions with estimates of 
jk 

the probability distributions of various attributes as explained above. Multiplying u 

Jk 

by the probabilities of all the probability branches traversed in going from the initial 
decision branch to the final outcome gives the contribution of this outcome to the 
expected utility of the decision. This value should be entered adjacent to each outcome. 
Adding up these figures at all outcomes stemming from an initial decision branch gives 
the expected utility of that branch. The branch with the largest expected utility is the 
one recommended by this procedure. 


5.5 SENSITIVITY ANALYSIS 

In forecasting future events, one often finds that the probability of one event is 
dependent, to some extent, on the occurrence or nonoccurrence of another event. For 
example, the introduction of direct broadcasts from satellites to home TV receivers 
may be dependent, in part, on the development of high energy nuclear-fueled power 
supplies suitable for satellites. It is very useful to have a measure of such depend- 
encies, for many reasons: 


1. A particular expert may be qualified to give a good estimate of the likelihood 
of dependent event A if conditioning event B occurs, and a good estimate of 
the likelihood of event A if B does not occur, but not be qualified to estimate 
the likelihood of B occurring. Some other expert may be qualified to esti- 
mate the likelihood of B but not the effects on A of the probability of B occur- 
ring. The sensitivity figures obtained from the first expert can be combined 
with the data provided by the second expert to get a better estimate of the 
probability of event A than either of them could provide alone. 
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2. If it is found that a desired event is sensitive to the occurrence or non- 
occurrence of a prior, conditioning event, steps can be taken to increase the 
likelihood that the conditioning event will occur. 

3. Through the use of sensitivity analysis it may be found that one event is 
dependent upon a second, which is dependent on a third, etc. By following 
chains of this sort, one may uncover connections between events that seem 
at first to be unrelated. If the first event on such a chain occurs, or does 
not occur, one can then estimate the effects on the probabilities of occur-* 
rence of the other events on the chain. For example, in recent years a 
change in the availability of anchovies off the coast of Chile reduced the 
availability of chicken feed (made from anchovies), which caused soy beans 
to be used for chicken feed, which reduced the availability of soy beans for 
cattle feed, forcing up the price of meat. If this chain of dependency had 
been known in advance, farmers could have been alerted to plant more soy- 
beans in time to prevent the severe soybean shortages that developed. 

4. If sensitivity analysis indicates that the probability of event A occurring 
does not depend much on whether event B occurs or does not occur, and if 
event A is the one the analyst is interested in, the analyst will know that he 
doesn't have to devote much effort to a careful estimate of the likelihood of 
B. On the other hand, if the probability of event A is strongly dependent on 
the occurrence of event B, the analyst may have to obtain a more accurate 
and complete estimate of the likelihood of B than he would otherwise have 
done. 

Sensitivity analysis, in the context of Delphi forecasts, is very simple. Respond- 
ents are simply asked to indicate which events, if any, would change the probabilities 
of occurrence, or date of occurrence, of a given event (called A) if they happen (or do 
not happen) at some previous date. The respondents are then asked to estimate the 
probability that event A will occur by the given date if one of these other events occurs 
by a specified previous date, and the probability that event A will occur try the given 
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date if the conditioning event does not occur by the specified date. The difference be- 
tween these two probabilities is the respondent's estimate of the sensitivity of event A 
to the conditioning event. 

After several respondents have made estimates of the sensitivity of an event A 
to a given conditioning event B, the results are tabulated. The median is used as the 
group estimate. The interquartile range (difference between the first and third quar- 
tiles) is also usually computed (see Appendix A. 2). A small interquartile range indi- 
cates that the respondents are in close agreement on the sensitivity; a large range 
indicates disagreement, and consequent uncertainty. 

Several points in the above procedure are deserving of further comment: 

1. The above procedure indicated that the respondents identify which events are 
sensitive to which other events. Sometimes the person creating the question- 
naires, rather than the respondent, decides which events should be con- 
sidered in this way, and formulates the questionnaire accordingly. It is also 
possible to ask respondents to evaluate the sensitivity of every event to every 
other event. Such a procedure is really a form of cross-impact analysis, 
which will be discussed in a subsequent section. 

2. The procedure outlined above asks respondents to estimate the probability of 
event A occurring by year y if event B has occurred by year w. The choice 
of y, and particularly w, is difficult. If w is made too close to y there may 
not be much time for B to affect the occurrence of A, even though the two 
events are closely related. Moreover, if w is made too close to the present 
the probability of event B occurring may be small. The estimates of the 
probability of event B occurring by year w, the probability of event A occur- 
ring by year y if event B does occur by year w, and the probability of event 
A occurring by year y if event B does not occur by year w, will change as a 
function of w. Ideally, the changes in these three quantities should be re- 
lated in such a way that the computed probability of event A occurring by year 
y will be the same for every value of w. In actuality, it is not likely that this 


5-16 



will occur. If the analyst and respondents have sufficient patience and time 
to estimate the probabilities and sensitivities for several different values of 
w, a useful check on the consistency of the estimates can be made. If the 
various computed values of the probability of event A occurring by year y 
come out about the same, the estimates are at least consistent. If the 
probabilities come out substantially different, depending upon the value of 
w, the estimates are inconsistent, and the reason for this inconsistency 
should be investigated. 


The probabilities and sensitivity factors are related as follows: 


P(A y ) = P(B W ) • P(Ay/B w ) * P(B w ) • P(Ay/B w ) 


where: P(A ) 

y 

P(B ) 
w 


= the probability that event A will occur by year y 
= the probability that event B will occur by year w 


P(A^/B^) = the probability that event A will occur by year y given that 


P(B ) 
w' 


event B occurs by year w 

= the probability that event B will not occur by year w 


P(Ay/B w > = the probability that event A will occur by year y given that 
event B does not occur by year w. 


Conditional probability is also discussed in Appendix A. 3, Cross Impact and Contin- 
gency Analysis. 


5.6 COMPUTER SIMULATION 


A digital computer can be useful in Delphi studies in at least four ways: 

1. Monte Carlo simulation can be done on a digital computer to perform a cross- 
impact analysis, as described below and in Paragraph 5.9. 

2. A somewhat different form of cross-impact analysis can be performed, as 
described below, using a computer program called KSIM. 
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3. The computer can be used as the medium through which the Delphi confer- 
ence itself takes place. Using teletype or similar remote access devices, 
participants can receive instructions, enter their data, and be informed of 

I 

questions, comments, and the results of previous rounds by the computer. 
The computer can serve as a convenient storage device, keep a record of 
all entries, and do the various arithmetic calculations required. Accounts 
of two such computer programs are given in References 8 and 28, and are 
summarized below. 

4. Computers are used in higher-level simulations. This approach is much 
more ambitious than is normally consistent with a typical Delphi exercise, 
but is potentially much more accurate. 

5.6.1 Monte Carlo Simulation 

As described in Paragraph 5.8, cross-impact analysis is a procedure in which 
the effects of the occurrence or nonoccurrence of certain events on the probabilities 
of other events are estimated. Starting from a set of initial estimates of the prob- 
abilities of various events and the impacts of these events on the probabilities of other 
events obtained from the Delphi participants, the object is to obtain a better estimate 
of the probabilities of the various events, allowing explicitly for the interactions. Or 
the object may be to see what effect changes in the probabilities of certain events 
would have on the probabilities of other events. The easiest way to obtain such a 
result is by Monte Carlo simulation. 

In Monte Carlo simulation, as applied to this problem, the computer starts with 
an initial event on the list. If the events are ordered chronologically, the computer 
would start with the first event. If two or more events affect each other's probabilities 
and there is no clear indication as to which would come first, one is selected at random. 
A random number between zero and one is selected by the computer, and compared 
with the probability of the event occurring. If the random number is less than the 
probability, the event is considered to have occurred. The probabilities of all events 
affected by that event are then changed to the value they would have if the event 
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occurred, and another event is considered. This procedure is repeated until all events 
have been considered. Tallies are then made by the computer of which events 
occurred, completing the first Monte Carlo trial. The whole procedure is repeated N 
times (usually about 1, 000), and the percentage of the N trials on which each event 
occurred is printed out by the computer. These updated percentages will be approxi- 
mately the same as the original percentages if the cross-impacts of the various factors 
are weak or tend to cancel out or. have been correctly included in the original prob- 
abilities, but may be quite different from the original percentages if the cross-impacts 
are strong and reinforcing in ways not fully considered in the original probability 
estimates. 

The decisionmakers, the government, or the outside world may take actions that 
change the probabilities of some events. To see what effect such changes would have 
on the probabilities of other events, all that is necessary is to change the initial prob- 
abilities of these first events, hold them at the desired value,, and rerun the simulation. 
Similarly, if an event is definitely known to have occurred (or if one wants to explore 
what would happen if it did occur),, its probability would be set and held at one. The 
changes in the probabilities of other events can then be tabulated. These changes are 
often not at all intuitive, because the many interrelationships in the cross -impact 
matrix often cause the probabilities to affect each other in a very complicated way. 

5.6.2 KSIM 

KSIM is the name of a computer program, or set of programs, which make it 
very easy for people with no programming experience to see how a system of interre- 
lated variables will change with time, given that they affect each other in certain ways. 

The first, and most important, step is for the key people familiar with the sys- 
tem to be modeled to get together and decide what are the significant variables, and 
to give at least a rough indication of how they affect each other. They create a matrix 
which somewhat resembles a cross-impact matrix, except that instead of the row and 
column headings being events, they are variables representing levels of activity or 
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quantity of some sort. For example, one variable might be the number of intercontin- 
ental telephone calls per year. Another might be the average cost of an intercontin- 
ental call. Others might be per capita income, the number of telephones per capita, 
level of international trade, etc. The participants must first decide what the variables 
are, then indicate, for every pair of variables, whether the first variable in the pair 
has a positive, negative, or zero influence on the second. If there is an influence, they 
must then decide whether the influence is weak, fairly strong, or very strong. Finally, 
the initial levels of the variables must be estimated by the participants, on a scale 
from zero (representing the lowest reasonable level of the variable that might be en- 
countered) to one (representing the highest possible level that might be encountered, 
within reason). The program takes this data and continually readjusts the computed 
level of each variable, allowing for the levels of the other variables and the cross - 
impact effect. It then plots out the results of this procedure as a function of time, 
showing the users the transient behavior and the general behavioral trends of the sys- 
tem. The simulation can then be rerun with different values for the initial levels or 
the cross-impact factors, to see how much effect such changes have. 

One of the variables that can be included in such an analysis is government or 
other outside intervention. For example, if government intervenes to hold down 
prices, such a constraint can be introduced into the model, and the effects on the other 
variables observed. 

It often turns out that the results are surprisingly insensitive to substantial 
errors or changes in most of the initial values of the variables and most of the cross- 
impact coefficients. The behavior of the model often doesn't change much in response 
to moderate efforts at governmental intervention, either. Certain factors are often 
found to dominate the behavior of the system, and even rather large changes in other 
factors shift the equilibrium only slightly. Results of this sort were discussed by J. 
Forrester in his book, "Urban Dynamics, " in which he pointed out that his models 
showed that most government efforts to improve the quality of housing in the cities were 
counterproductive or ineffective, and this seemed to correspond to actuality. He stated 
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that interventions in complex systems often lead to results which are entirely at odds 
with the initial expectations. Any complex system defines an integrity of its own and 
strongly resists external changes, a fact well understood by ecologists. When com- 
plex systems change they seldom change continuously but rather flip suddenly into an 
entirely new configuration. Such results are of great interest, because they can keep 
large sums of money from being spent on programs that will be ineffective. This situa- 
tion also can increase the confidence of the user in the results he obtains, since some 
error in one or several inputs is not likely to change the final state, or even the tran- 
sient behavior, of the system very much. The factors which are critical, if any, can 
be identified by experiment, and an effort can be made to get better data on those. 

The number of cross-impact cells that must be filled in by the participants is 
proportional to the square of the number of variables in the set being considered, so 
there is a strong incentive to keep the number of variables down. As a result, the 
model created will have to be highly aggregated, in most cases. Nevertheless, the 
program can produce some very useful and unexpected insights into the behavior of a 
system. After setting up and working with such a model, the participants will usually 
have a much better appreciation of what factors dominate the behavior of the system, 
and why, than they could obtain in any other way. 

5.6.3 The Computer as a Medium 

The Delphi exercise is an attempt to obtain the benefits of a conference without 
some of its disadvantages. One means by which a Delphi exercise can be implemented 
is to call the participants to a common meeting place, but to control the interaction 
between them by artificial means, so that individual participants do not control the 
participation of others through the force of their personalities. Such a methodology 
has the disadvantages of all conferences - the difficulty of finding a meeting time that 
is convenient for all participants, the travel costs (both time and money) for the partic- 
ipants who are distant from the meeting place, etc. The alternate method of procedure 
has usually been to conduct the proceedings by mailed questionnaires. This is much 
more convenient for the participants, but is slow. A Delphi conference could be run 
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by telephone, but such a procedure is rather slow and awkward, and there are time 
zone problems if the participants are spread out geographically. An interesting and 
useful alternative, therefore, is conducting the conference by computer. In Reference 
29, Vallee described a computer program named FORUM that permits people to 
exchange ideas and information, make decisions, participate in making forecasts, etc. 
The users do not require training in the use of a particular computer language, and can 
go through an entire conference without learning a single command. They must, of 
course, use a computer terminal, such as a teletype, but no special or control charac- 
ters are used. FORUM not only has the ability to administer hundreds of question- 
naires simultaneously, but it also allows a completely open, unstructured conferencing 
style. It performs the routine functions of administering the inquiry, providing instruc- 
tions to participants, leading them through the agenda, posing questions, storing and 
processing answers, presenting summaries of results, etc. In addition, it allows the 
participants to ask questions, make comments, permits deviations from the original 
agenda, and is quite flexible. One of the valuable by-products is a written record of 
the discussion and context in which decisions were reached, which is often not available 
with conventional conference techniques. 

The FORUM program was written for a PDP 10 computer operating under the 10X 
operating system. It could, of course, be modified or rewritten for use on other com- 
puters. 

In Reference 8, Turoff describes a somewhat similar program he wrote and de- 
bugged in about three man-months of effort. It is written in XBASIC for the Univac 
1108. An improved version was later written in 3 months' time by a professional 
programmer. Turoff conducted a Delphi exercise on the usefulness of his original 
program, and asked ideas for ways it could be improved. The results of the exercise 
are given in the same reference. 

One of the advantages of using a program written in a simple language such as 
XBASIC is that it can easily be modified by the Delphi conference monitor if he, or the 
participants in the Delphi conference, desire to do so. 
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5.6.4 Higher-Order Simulation Models 

The KSIM simulation described previously is a simplified, relatively easy-to- 

use mechanism for estimating the effects of various variables on each other. Although 

the results it produces are often of considerable interest, the idea that the effect of one 

variable on another can be represented by a single number is a gross simplification, 

and, one may fear, in most cases an oversimplification. As Julius Kane says in his 
30 

paper on KSIM, "For greater realism we have begun consideration of cascaded 
models of the variety described. This permits the interactions themselves to be func- 
tions of the state of the system. Clearly A need not always be B's friend. A's attitude 
towards B can be conditional on the relative status of their difference A-B or perhaps 
depend upon the state of a third individual C. Perhaps much more important is the 
following refinement. Often we sense not a variable tut rather changes in a variable. 
Our response to environment has much this character. Whether our locale is good or 
bad we quickly become acclimatized to it and then become sensitive only to gross 
changes. Such derivative interaction is very important and is the subject of other 
papers (Kane, Vertinsky, and Thompson). " 

It may be added that there are also often time delays, of various magnitudes, 
that enter into the interactions between the variables. 

Estimating the interactions between the variables in this more detailed but real- 
istic way is a substantial effort, and thus it cannot really be considered that creating a 
higher-level simulation would be a way of using a computer to support a Delphi exer- 
cise. On the contrary, the Delphi exercise would probably be used to support such a 
higher-level simulation effort, identifying the variables to be simulated and the level of 
aggregation for each variable, and defining the cross -impact functions. J. Forrester 
has used such simulations (though without benefit of Delphi techniques, insofar as is 
known) in his work on Industrial Dynamics, Urban Dynamics, and World Dynamics. 
Even more elaborate simulations have been made in the economic field, in which 
the U. S. economy is divided into several dozen or several hundred sectors, and 
the differential equations linking each sector to each other sector are estimated 
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and modeled. The effort required is large but the results are correspondingly im- 
proved. Our society consists of many interacting multiloop feedback systems, some 
with positive feedback, some with negative. With time and effort we can make pretty 
fair estimates on how each little link of the whole apparatus works. Human beings, 
rather than computers, have to do that part of the job. When it comes to figuring out 
how all these feedbacks and time delays affect each other, however, only a computer 
can cope with the problem. Forrester first, and most successfully, demonstrated the 
utility of this approach when he showed why some firms were subject to violent oscil- 
lations in their level of business activity whereas others were relatively stable. He 
did this by the method outlined above, and described in his book "Industrial Dynamics. " 

The level of detail in the model need not be extremely great in order to produce 
useful results. In his book "World Dynamics, " for example, he describes the results 
of his studies on a model containing only five variables (population, pollution, food 
supply, industrial development, and raw materials), all aggregated on a global basis. 
The whole computer program consists of only 127 lines of computer code. 

The interactions simulated by Forrester are all of the types called "trend to 
trend" by Helmer in Reference 31. The same applies to KSIM. They are, however, 
more realistic than Helmer' s trend to trend interactions. Helmer attempted to define 
the interactions in terms of a matrix which shows the effects on trend j if trend i de- 
parts from its initially predicted value. In reality, the interactions are much more 
complicated than this, as mentioned above in connection with KSIM. Helmer 's approach 
has the advantage over Forrester's simulations and KSIM, however, in explicitly model- 
ing interactions between events and events, events and trends, and trends and events, 
in addition to the interactions between trends and trends. Forrester did not model the 
effects of trends on events or events on events. He did model the effects erf events on 
trends, in essence, by changing the values of certain coefficients in his simulation at 
particular future times, to represent changes in government policy, people's values, 
etc. In summary, these higher-order simulation models are the most realistic and 
accurate way we have, at present, of predicting the complicated interactions of our 
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society. They will almost certainly be used increasingly, and become increasingly 
accurate. For example, it is easy to enrich and extend KSIM or Forrester's detailed 
trend to trend simulations by including the effects of trends on events and events on 
events. Such higher-order simulations are, however, major efforts in their own right. 

5. 7 REITERATION OF PROBABILITY ESTIMATES 

It has been shown that the consensus judgment of a group using the Delphi tech- 

7 

nique improves with feedback. There are numerous variations in the way informa- 
tion can be gathered and fed back and successive rounds handled. Reference 32, for 
example, recommends a procedure which is rather inapt for estimating a probability 
density function using the Delphi technique. In that example, which had to do with 
estimating the probability distribution of the thrust of a jet engine, each of five ex- 
perts, upon being questioned alone, gave five values of thrust which he could discrimi- 
nate between on a probability basis. He was subsequently given a thrust value picked 
randomly from a list of 11 values (which he could presumably not discriminate between) 
and asked to estimate its probability. The longer list was a compilation of the thrust 
values mentioned by the five experts when questioned individually, and was apparently 
not revealed to the participants. In a situation like this, one person might respond with 
a low probability value, since he might figure that other, nearby values of thrust might 
be just as likely, whereas another might respond with a high number, since he might 
figure that the thrust he is being asked about is in the vicinity of the high probability 
region of performance. After three rounds on this one value of thrust, there were 
another three rounds on the next randomly selected value from the list, so that a total 
of 33 rounds was required to complete the analysis of this one question. If more care 
had been taken in the design of this procedure, the amount of. time required of the 
participants could have been reduced substantially and much more information obtained. 

A better procedure in a case like this would be to work with cumulative prob- 
ability distributions, rather than with probability densities. It is easier for a person 
to think that an engine is 90 percent likely to have more than 31, 000 pounds of thrust 
and 10 percent likely to have more than 37, 000 pounds of thrust than to estimate the 
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probability of its having exactly 31, 000 pounds of thrust, particularly when he does not 
know what the alternatives are he will be asked about. Furthermore, by asking each 
expert on the same round where he would put the 90 percent, 75 percent, 50 percent, 

25 percent, and 10 percent cumulative probability values, the total number of rounds 
could be reduced to three or four, instead of 33. Once consensus has been reached on 
the cumulative probability distribution, it can be converted to a density function, if 
desired. 

After the first round of answers have been collected, the participants are given 
the median and quartile results for each question, along with data on what the partici- 
pant's own answer was to each question. If enough people rate themselves as relatively 
knowledgeable on a question so that they form a subgroup of adequate size (at least 7, 
with 12 or more desirable), feedback should be based on the answers only of this 
knowledgeable subgroup, and on such questions the participants in the less knowledge- 
able subgroups can be instructed to skip the question in subsequent rounds unless they 
have some comment they want to make. For example, they may know of some reason 
why the arguments supplied for or against a particular estimate are invalid, or they 
may have some reason for making a particular estimate that none of the people who 
rate themselves more highly, has mentioned. Participants going on to subsequent 
rounds are asked to revise their estimates if they wish to and to supply reasons for 
their estimates if they fall outside the interquartile range. Starting on the third round, 
when such reasons are fed back to the participants, the participants are also asked 
to refute any reasoning they believe to be faulty. 

There is some disagreement about the extent to which participants should be 
urged to answer all questions, particularly on the first round. For example, Refer- 
ence 33 says "Panelists — particularly those with technical backgrounds — must be con- 
vinced that judgments often have to.be made about issues before all facets of the prob- 
lems have been researched and analyzed to the extent they would like. (For these situ- 
ations they must be persuaded that their subjective judgments may be a decisionmaker's 
most valuable source of information. )" However, that same reference, on the 
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following page, says "If a multidisciplinary approach is desired, respondents should 
be encouraged to consider all items but to make estimates only on those scaled de- 
scriptive phrases with which he feels comfortable. For example, in these exercises 
it was helpful when respondents indicated their familiarity with a specialized area or 
the importance of an item even though they did not make probability estimates. " 

Four rounds are generally used in a Delphi exercise, in order to give participants 
a chance to respond to and refute arguments raised for particular estimates during the 
second and third rounds. If new arguments or refutations are advanced during the 
fourth round, or if participants are still changing their positions appreciably in this 
round, additional rounds can be added until no further substantial change appears likely. 

The final output of the exercise is usually the median and quartile points for each 
question, based on the last round results for the subgroups consisting of self-rated ex- 
perts. If the population of such subgroups is small, the results are based on the esti- 
mates of the whole group of participants. If the group seems to divide itself into two 
or more relatively large factions clustered closely about substantially different 
estimates, this should be noted. The arguments for and against the various estimates 
should also be preserved. 

During the course of the exercise, some participants may ask that additional 
questions be added to the list being considered. The answers to such supplementary 
questions may be helpful in answering questions on the original list. Such questions 
can be submitted to the other members of the group for a vote on the usefulness of add- 
ing them to the list, or may be added to the list at the discretion of the team conducting 
the exercise. 

Reference 33 recommends letting the panelists decide through their suggestions 
and evaluations what items should be considered. The criteria for retaining an item for 
further evaluation should be made clear at the outset of the exercise. 

Some Delphi exercises have been conducted with different participants on each 
round. No comparative studies have been published on the benefits and disadvantages 


5-27 



of this procedure as compared with using the same participants for all rounds. It is 
much more customary to use the same participants on all rounds, except, as has been 
mentioned, that participants rating themselves low on expertise on certain questions 
are sometimes asked to refrain from answering those questions. 

Other helpful hints mentioned in Reference 33 for use during reiteration of the 
probability estimates are noted below: 

1. Interpersonal techniques, such as interviews and seminars, should be inter- 
spersed with the rounds of questionnaires and information feedback. 

2. The source of a suggested item should be identified (for example, panel 
member number and basic biographical information), taking care not to 
compromise the anonymity of specific inputs. 

3. Standardized scaled measures should be available to a respondent so that he 
can qualify his response to specific questions. Such measures are relative 
competence in a technical area, familiarity with a geographical region, or 
confidence in an estimate. 

4. If the exercise is conducted by mail, a definite date on which the question- 
naires are to be completed should be specified. 

5. Personal comments and arguments submitted by respondents should be part 
of the information feedback. 

5.8 CROSS-IMPACT ANALYSIS 

Cross-impact analysis is a method of taking into account the interrelationships of 
a set of events in assessing the individual probabilities of those events. It is applied in 
two ways: 

1. Cross-impact analysis is used in an attempt to improve the estimates of 
probabilities of future events, based on an initial set of estimated prob- 
abilities and a matrix of cross -impact effects. 
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2. It is also used to find out what happens to the probabilities of certain events 
if other events occur. This is useful in two ways: 

a. It may be possible to increase or decrease the probabilities of certain 
events. The cross -impact analysis shows what effect this would have 
on the probabilities of other events. 

b. With the passage of time, some of the events in the matrix will occur, 
or not occur. The results of the analysis can then be updated by simply 
changing the probabilities of those events to one or zero, as the case 
may be, and rerunning the cross -impact analysis. Such a procedure is 
much faster and cheaper (though probably not as accurate) as rerunning 
the whole study by obtaining new probability estimates from a panel of 
experts. 

Several different procedures have been reported in the literature for obtaining 
the cross-impact matrix and performing the analysis. None of them can be completely 
justified mathematically, and the procedures which come closest to possessing mathe- 
matical consistency are the most awkward and tedious to apply. A discussion of the 
differences between the various procedures which have been suggested, and their short- 
comings, is given in Appendix A. The procedure outlined below is intended to combine 
the better features of several approaches, plus some simplifications which appear 
obvious but have not been reported in the literature. The major reason for the mathe- 
matical problems is that although the cross-impact matrix contains considerably more 
information than a simple list of estimated probabilities does, it still does not contain 
enough information to uniquely specify all. the joint probabilities. It does, however, 
contain enough information to be capable of being inconsistent with itself and with 
a list of estimated probabilities. Checking for, and removing, the inconsistencies 
poses difficulties, and may or may not improve the quality of the estimates produced. 
Therefore, the results of the cross-impact analysis should be used with caution. 
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The recommended procedure is as follows: 


1. Obtain agreement on a list of events whose probabilities are to be estimated. 
In some situations the list may be prepared in advance and presented to the 
participants; in others the participants in the study will create the list. In 
any case, there will probably be revisions to the definitions of the events, 
and possibly events added or deleted from the list, on the basis of the com- 
ments by the participants. 

The list should not be too long, because the number of cells in the cross - 
impact matrix grows approximately as the square of the number of events 
considered. 

2. The participants should estimate the probabilities of occurrence of each 
event. 

If possible, the probabilities should be in the form of numbers, on a scale of 0 to 
1 or 0 to 100 percent. If this is confusing to the participants, they can be given a copy 

of Table 5-1 which shows the equivalence between English phrases and probability 

. 33 

values. 


Table 5-1. Probability Equivalents of Common Phrases 


Phrase 

Probability Value 

Odds in Favor 
of Event 

Very Unlikely 

0. 10 

1 to 9 

Rather Unlikely 

0. 25 

1 to 3 

Toss Up 

0. 50 

1 to 1 

Good Chance 

0. 75 

3 to 1 

Highly Probable 

0.90 

9 to 1 


Table 5-2 gives an example of a list of events and their probability values. The 
list would usually be longer in a real Delphi exercise. 
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Table 5-2. Events and Their Estimated Probabilities 


Estimated Odds In 

Event Probability Favor 


El. 

Development of 100 kW ERP satellites by 1980 

.75 

3:1 

E2. 

Considerable commercial use of optical waveguides 
by 1980 

.25 

1:3 

E3. 

Direct satellite-to-home TV becomes common between 
1980 and 1990 

. 50 

1:1 

E4. 

Pocket-size portable telephones become widespread 
between 1980 and 1990 

. 50 

1:1 

E5. 

Picture phone service or equivalent, becomes widespread 
between 1980 and 1990 

. 25 

1:3 

E6. 

Most remote access computer terminals have graphic 
display capability by 1990 

.30 

3:7 


3. The participants should then fill in two cross-impact matrices. 

The first 


matrix gives their estimate of effect on the probability of each event if some 
specified event is certain to occur, and the second matrix gives the effect on 
the probability if the specified event is not certain to occur. Figures 5-4 and 
5-5 are examples of such matrices. If the occurrence or nonoccurrence of 
the specified event has no effect on the probability of occurrence of the other 
event, the cell in each of the two matrices at the row corresponding to the 
specified event and the column corresponding to the other event should be 
left blank or have a zero entered. If the occurrence of the specified influenc- 
ing event increases the probability of the other event, a positive number 
greater than one should be entered in the first matrix. The magnitude of the 
number signifies how much the odds in favor of the event will be multiplied 
if the influencing event occurs. For example, if a four is entered, an event 
which had odds of 1:1 (probability 0. 5) would be given odds of occurring of 4:1 
(probability 0.8) if the influencing event occurs. If the odds were originally 
4:1, the odds would be raised to 16:1 (probability 0.94) if the influencing 
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event occurs. If the odds were originally 1:20 (probability 0.0475) they 
would be raised to 4:20 or 1:5 (probability 0.167). 

If the participant believes that the occurrence of the influencing event would 
reduce the probability of the other event, he should enter a negative number 
in the corresponding cells of the first matrix. The magnitude of the number 
will then indicate the factor by which the odds will be divided. For example, 
if a -4 is entered and the odds in favor of the event were originally 16:1, 
they would be reduced to 4:1 if the influencing event occurred. If the odds 
were originally 1:5, they would be reduced to 1:20. 

The second matrix should be filled in similarly, except that here the numbers 
refer to the effects on the probability of an event if the influencing event 
definitely does not occur. The numbers entered in the two matrices will 
often be different in magnitude. For example, if optical waveguide comes 
into widespread use the likelihood of picture phone service becoming common- 
place may be increased substantially, but if optical waveguide doesn't come 
into widespread use the likelihood of picture phone service becoming common- 
place may not decrease much. Other fairly likely developments may make 
picture phone service practical even without optical waveguide. 

4. The responses produced by the various participants are then collected, and 
the median and interquartile values for the probabilities and impact factors 
are tabulated. All three values are fed back to the participants for use in 
subsequent rounds. After the last round the median values are used for 
further processing. 

5. The next steps can most conveniently be done by a computer, if one is availa- 
ble. If not, they can be done manually. 

a. Scan the two matrices of median impacts and find which events, if any, 
do not affect the probabilities of any other events and are not affected by 
the probabilities of any other events. Event E4 in Figures 5-4 and 5-5 


5-32 
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Figure 5-4. Matrix Showing Impact of Occurrences 



Then this would be the factor used to adjust 
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Figure 5-5. Matrix Showing Impact of Nonoccurrences 
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is such an event. All of the numbers in column E4 are zeroes in both 
matrices, showing that E4 is not affected by the occurrence or non- 
occurrence of any other event. All the numbers in row E4 in both 
matrices are zeroes, showing that E4 does not affect the probability of 
any other event. When events such as E4 are identified, they should be 
removed from the matrix and noted separately, since they are independ- 
ent of the other events in the matrix. 

b. If the matrix consists of independent groups of events which interact 
within the group, but not between groups, the matrix should be broken 
up into submatrices corresponding to these groups. This greatly 
simplifies the processing, presentation, and interpretation of data about 
them. In the example given herein, events El and E3 are related, and 
events E2, E5, and E6 are related. These can be split apart as shown 
in Figures 5-6 and 5-7. A method for splitting the matrices in this 
manner is given in Appendix A. 

Although it seems obvious that simplifying the matrices in this manner 
should clarify and simplify further work on them, this step has not been 
mentioned in previous descriptions of the cross -impact procedure. 

c. Within each submatrix, the events that are not affected by any other 
event should be listed first, then the events which are affected by only 
one other event, etc. 

6. The median, initial probabilities and cross-impact values are used as inputs 
to a Monte Carlo computer simulation which uses them to compute revised 
probabilities. The procedure is explained in Paragraph 5.6. During each 
Monte Carlo trial run, the events which are not influenced by any other event 
should be simulated first, then the events which are only influenced by one 
other event, etc. If several events influence each other’s probabilities, 
their order of simulation should be random. 
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Figure 5-6. Simplified Matrices Showing Impact of Occurrences 
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After the computer makes several hundred or several thousand Monte Carlo 
trial runs, the percentage of trials during which each event occurred will be 
printed out. These percentages are the revised estimates of the probabilities. 
If the participants properly allowed for all the interactions when making their 
initial estimates of the probabilities of the various events, the revised set 
should agree fairly closely with the initial set. If there is disagreement 
between the initial values and the revised values, the cause may be one or 
more of the following reasons: 

a. The revised values may be better estimates than the initial probability 
values, because the cross-impact effects have been taken into account 
more fully. There is no experimental evidence at the present time that 
would support (or refute) this interpretation, however. 

b. The cause may be inconsistencies within the cross-impact matrices and 
between the cross-impact matrices and the data. Even if there are in- 
consistencies, or perhaps especially if there are inconsistencies, the 
revised probability estimates may be better than the original estimates. 
However, it is possible they may be worse. As mentioned above, 
although the cross-impact method is used, it has never been validated. 
Validating it is difficult because it is usually used to estimate the prob- 
ability of occurrence of unique events. 

c. Statistical noise inherent in the Monte Carlo process causes some 
scatter in the results of Monte Carlo simulation. This noise is inversely 
proportional to the square root of the number of Monte Carlo trials 
made. Appendix A contains data on the expected magnitude of this effect 
as a function of the number of trials. 

5.9 CROSS-SUPPORT ANALYSIS 

The purpose of doing studies is to provide information which leads to decisions 
and action. One cannot know what decisions to make unless one knows what goals one 
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is trying to achieve, and the relative importance of different goals. Some goals may 
be of only moderate importance in themselves, but achievement of these goals may 
contribute significantly toward the achievement of other goals. Such goals should get 
more support than goals of equal importance which do not support other goals, or 
which interfere with the achievement of other goals. Cross-support analysis provides 
an orderly method of assessing the importance of various goals, considering both their 
inherent desirability and their interactive effects. The procedure outlined herein is 
based on Reference 23. 

Step 1 . The initial step is to list the major goals which are to be achieved. The 
list may be provided by the organization sponsoring the study, or it may be produced 
by the participants. 

Next, the importance of the individual goals should be assessed based on their 
relative importance. This is based on the criticality of the goal and the degree of 
advancement which one hopes to make in the near future.. The mechanism for assign- 
ing importance is to distribute a total of 100 points among the goals in proportion to 
their estimated importance. In a study bearing on communications, the goals of com- 
munication policy, and their relative importance, might be as shown in Table 5-3. 

Table 5-3. Goals of Improved Communications 


Improve educational services 15 

Provide more employment 15 

Increase exports 15 

Reduce costs 10 

Improve public access to news and information 10 

Improve convenience 10 

Improve safety, police and emergency services 10 

Improve privacy 5 

Improve security 5 

Provide greater variety of entertainment 5 


100 


5-37 



The judgments on the relative weights of the various goals should be based on 
their relative desirability, not on factors such as technological costs, capital invest- 
ments, amount of budget allocated to each goal in previous years, political factors, etc. 

Continuing with the analysis, subtargets under each major goal are then identified 
and weighted in the same manner. The total of the weights of the subtargets under each 
major goal is constrained to the weight of the major goal, so that the total weights of 
all subtargets will equal 100 points. For example, the following subtargets might be 
identified and weighted under the major goal of convenience: 


Major goals and subtargets 

Major goal weight 

Subtarget weights 

CONVENIENCE 

10 


Availability 


3 

Portability 


2 

Reliability 


2 

Usability by handicapped people 

1 

Ruggedness 


1 

Fidelity and clarity 


1 


Step 2 . The major goals and their associated subtargets, together with their 
respective weights, are transcribed on the left-hand margin of a matrix. Figure 5-8 
illustrates an abbreviated version of such a matrix, without any subtargets shown for 
any of the goals except convenience. The subtarget weights from step 1 are also to be 
listed under the column headed OW (Original Weight). To facilitate subsequent manipu- 
lation of the subtarget-to-subtarget matrix, it is suggested that these be called primary 
subtargets. 

Step 3 . The major goals and subtargets and their associated weights are also 
transcribed across the top of the matrix. It is suggested that these be called comple- 
mentary subtargets to distinguish them from the primary subtargets in step 2. At the 
right end of the matrix also provide columns labeled TCS (Total Cross Support), APA 
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Figure 5-8. Example of Cross-Support Analysis Matrix 


(Additional Points Added), OW (Original Weight), NW (New Weight), and NNW (New 
Normalized Weight). 

Step 4 . Having constructed the framework of the goal-to-goal cross -support 
matrix, the next step in the methodology is to determine the degree to which achieve- 
ment of one subtarget (goal) contributes to the achievement of all others. This is done 
by considering the effect one subtarget has on each of the others, and then summing 
these separate contributions to obtain a total. The effect that one subtarget may have 
can either be beneficial (positive) or detrimental (negative). For example, reducing 
costs may tend to reduce employment but increase exports. 

Each cell of the matrix should be divided by a diagonal line. Above the diagonal 
line the participant filling in the matrix should enter an H, M, or L, if he thinks the 
contribution of the primary subtarget to the complementary subtarget is positive and 
high, medium, or low, respectively. If he thinks the contribution is negligible, he 
should leave the cell blank. If he thinks the contribution is negative, he should enter 
-H, -M, or -L. 

Numeric values will be associated with these letters as follows: 

H = 4 
M = 2 
L = 1 

Multiply the numeric equivalent of the letter in the upper part of the cell by the weight 
of the complementary subtarget and enter the product in the lower part of the cell. If 
the upper part of the cell is blank, the lower part should be, also. 

Step 5 . When the lower part of all applicable cells are complete, sum horizon- 
tally and place the total in the column headed Total Cross Support (TCS). Sum up all 
the entries in the TCS column and place the total at the bottom. 

Step 6 . Apportion 100 points to the Additional Points Added column, giving each 
row a number of additional points equal to its percentage of the total TCS score. Add 
up the APA column. The total should be, of course, 100 points. 
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The decision to award 100 additional points for cross-support effects, rather 
than 50 points or some other number, is quite arbitrary. No justification has appeared 
for it in the literature. It would probably be desirable to let the participants decide on 
what this number should be, rather than always using 100 points. For example, if 
only one of the subgoals supported any other, the subgoal doing the supporting would 
get a normalized new score larger than all other subgoals combined. Such a result 
would be highly unrealistic. This is admittedly an extreme example, but one which 
reveals that in situations where there is little cross-support the total points awarded 
for cross support should be less than 100. 

Step 7 . On each row, add the entry in the Additional Points Awarded column to 
the entry in the Original Weight column, and enter the total in the New Weight column. 
The New Weight column should add up to 200 points. 

Step 8 . Finally, normalize the weights by dividing each entry in the New Weight 
column by 2, and entering the result in the Normalized New Weight column. The 
entries in the NNW column are the revised weights of the various subgoals, allowing 
for cross-support effects. Add up the values for the subtargets under each goal to get 
the weight to attach to each goal. 

Using the above procedure helps assure that all important goals and subgoals will 
be considered and their effects assessed. It also. preserves a historical record estab- 
lishing that various conflicting interests were considered and, insofar as possible, 
weighted according to their importance. Use of this sort of approach also increases 
the chances that decisions taken in different years will be more consistent than they 
would be in the absence of such a framework. 

As can be seen from Figure 5-8, the new normalized weight produced by the 
cross-support matrix method can differ substantially from the original weights assigned 
to the goals. For example, the subgoal of fidelity and clarity was initially assigned a 
weight of 1 point but the new normalized weight for it was -5. 1 points. This occurred 
because it was felt that any developments which increase fidelity and clarity are likely 
to have a high negative impact on costs and portability. It is also interesting to note 
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that the goals of privacy and security both nearly doubled weight from 5 to 8. 8 because 
they strongly support each other. 

Of course, the example presented herein is purely hypothetical, and no signifi- 
cance should be attached to the weights assigned or the results obtained. A great deal 
of judgment must be used in assigning the cross-support factors. For instance, in this 
example the effect of costs on employment was judged to be negative medium, and the 
effect of fidelity and clarity on employment was judged positive low. But stereo hi-fi 
equipment, which costs nearly twice as much as monaural and is less portable, has 
been selling well, boosting employment. 

5.10 THE POTENTIAL AND EVOLVING STATUS OF DELPHI 

The Delphi technique was originated relatively recently, in 1948, and did not 

34 

come into general use until 1964. As mentioned in Paragraph 5. 1, it is still evolv- 
ing in most of its detailed procedures. Nevertheless, it is possible to list its basic 
assumptions, outline the types of problems in which it is useful, and attempt to evalu- 
ate its growth potential. 

The basic assumptions of the Delphi method are the following: 

1. N heads are better than one. In other words, a group of people will have 
more ideas, think of a problem from more angles, and usually will do better 
at estimating partially known factual data, than a single individual of the 
group would do. There is a subassumption here that the single individual 
who would be asked the questions if the Delphi method were not used is not 
outstandingly more informed about the problem than the other members of 
the group are. 

2. Conventional conference techniques for eliciting a group consensus are beset 
with the following problems: . 

a. Many people don’t like to abandon a position once they have proclaimed 
it in public, even if they are later shown evidence which would other- 
wise make them change their mind. 
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b. Conferences are often dominated by the views of one vociferous or 
prestigious individual. 

c. Conference discussion often meanders and in some cases never does 
arrive at a consensus. 

d. Committees often fail to make their assumptions and reasoning explicit, 
since their findings are obtained through bargaining. 

e. A conventional conference requires considerable time from the partici- 
pants, particularly if some of them have to travel a long way to attend. 

On the other hand, conferences do provide a flexible mechanism for exchanging 
and stimulating thoughts and ideas. The Delphi procedure attempts to retain and en- 
courage this, insofar as possible, by permitting participants to explain their reasons 
if they are not in the center 50 percent, by preserving anonymity with regard to who 
made which comment, and by feeding back to participants the median and interquartile 
range from the previous round. The format of the questions deters the participants 
from going off on tangents and permits them to supply their ideas relatively rapidly 
and at their own convenience. The Delphi technique also helps individuals in separate 
fields to establish meaningful constructive cooperation. If all the participants have 
convenient access to remote computer terminals which are netted together, a Delphi 
conference can be called on short notice, and can continue for as short or long a time 
as is appropriate. Furthermore, larger numbers of people can participate at such a 
Delphi conference than would be practical at a live conference. 

Delphi techniques seem to be most useful for the following types of problems; 

1. Making numeric estimates of unknown quantities, whether the quantity be the 
time until some new development is introduced, the probability of a particu- 
lar project or medical treatment being successful, the probable performance 
of some new device, or the number of doctors in Baltimore in 1880. The 
emphasis on numeric estimates arises because the most feasible and effec- 
tive feedback seems to be the median and interquartile range of the estimates 
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produced by members of the panel. If estimates are not numeric, or at 
least rankable, it is difficult or impossible to define the median and the 
interquartile range. For example, if a group of economists are asked what 
action the government should take in the economic field at the present time, 
and each gives an entirely different answer, ranging from price controls to 
raising taxes to raising interest rates, no median can be defined. 

2. Problems of a brainstorming sort. In spite of the impossibility of feeding 
back median and interquartile information as noted above, a Delphi confer- 
ence can produce many ideas for possible solutions to a problem. A regular 
brainstorming conference can do this also, but some types of people may be 
too inhibited to present their tentative and undeveloped ideas at a conven- 
tional conference. The anonymity of the Delphi conference is a help to such 
people. 

3. Problems involving judgment, opinion and goal selection. If the participants 
are chosen properly, a Delphi exercise can increase the probability that no 
significant viewpoints or factors are overlooked. 

Delphi exercises, when properly managed, have been found to be a highly moti- 
vating environment for the respondents. Quoting from Reference 7, "The feedback, if 
the group of experts involved is mutually self-respecting, can be novel and interesting 
to all. The use of systematic procedures lends an air of objectivity to the outcomes 
that may or may not be spurious, but which is at least reassuring. . . The experience 
of many practitioners has shown that the results of a Delphi exercise are subject to 
greater acceptance on the part of the group than are the consensuses arrived at by 
more direct forms of interaction. All of these features of a Delphi exercise are desir- 
able, especially if the exercise is conducted in the context of policy formulation where 
group acceptance is an important consideration." 
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Reference 35 sums up the potential and evolving status of Delphi even better: 

"Further experimental work is needed. This includes using the Delphi technique 
in conjunction with in-depth interviews, structured conferences, and operational gam- 
ing. .. But, imperfect as it is, the Delphi process or some further modification 
appears to be one of the most promising approaches under 
gation of problems with a high social and political content, 
allocate resources rationally and to force explicit thinking 
benefits, it offers a hope of introducing cost-effectiveness 


development for the investi- 
Be cause it can be used to 
about the measurement of 
thinking into these problems." 
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SECTION 6 - EXAMPLES OF DELPHI METHOD APPLICATIONS 


This section presents four examples of application of the Delphi method. They 

have been selected to illustrate the flexibility of Delphi and types of results which may 

16 

be achieved. The Japanese scientific and technological survey, in particular, in- 
cluded a number of communication items, many of which are indicative of communi- 
cations developments that also may be expected to occur in the United States. These 
communication items are included to provide additional basis for selection of subject 

areas of inquiry which may be undertaken in a NASA Delphi forecast of telecommuni- 

36 

cations developments. The example from Quade and Boucher contrasts simple and 

more complex Delphi applications and serves to clarify the principles of the Delphi 

33 

technique. The University of Michigan Sea Grant inquiry critically evaluated 

Delphi techniques in an operational application to long-range research and planning 

efforts for water resources management. The Sea Grant Delphi is a progressive 

study with a number of innovative adaptations worthy of consideration in further 

13 

Delphi applications. The TRW Probe II Delphi study illustrates industry belief in 
and commitment to Delphi as an aid in R&D planning and long-range forecasting to 
help management improve its decisionmaking processes by providing immediate in- 
formation on probable and significant future trends. 

6.1 JAPAN TECHNO-ECONOMICS SOCIETY PUBLICATION 

6.1.1 Introduction 

In 1971 Japan's Science and Technology Agency made public the results of a 
forecast survey on scientific and technological developments in the 30 years up to 
2000 A. D. This survey used Delphi techniques involving 4000 experts on 620 develop- 
ment subjects. The report of the survey, published by the Japan Techno-Economics 
Society, was entitled "Science and Technology Development up to 2000 A.D." 

A basis for the study was the assumption that science and technology will lead 
the way in establishing the future society and economy of Japan. Prior to the actual 
survey, a general study group formulated subject areas, questioning methods, format 
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of results, etc. , with the assistance of five sectional groups, covering the areas: 
social development, information, medical care and health, food and agriculture, 
industry and resources. The forecast sought to probe future social and economic 
needs, the technology to meet such needs, and attendant technological consequences. 

A number of guiding principles were adopted for the survey by recognition that: 

1. Changes in social and economic needs (values) impact technological 
forecasting. 

2. A comprehensive national view of all science and technology fields must 
be taken. 

3. Both normative and exploratory approaches must be adopted. 

4. Emphasis must be placed on identification of tasks of prime urgency, 
i. e. , priorities must be established by evaluation of the relative im- 
portance of the various developments. 

6.1.2 Planning the Survey 

A 30-year period for the survey was selected as a balance between allowing a 
sufficient interval to take account of discontinuous technological changes or "break- 
throughs" and yet not extending over so long a period that foreseeing changes in needs 
due to changes in people's concept of values would be of exceeding difficulty. Since 
the field of science and technology is so broad, adequate handling could be assured 
only by dividing the field into the five areas mentioned in the previous paragraph. 

In selecting a technique for forecasting, the projected scope of the effort in subject 
matter, the considerable period of 30 years, a primary desire to evaluate the urgency 
of individual tasks, and the wide range of persons in scientific and technical develop- 
ment - all led to a decision to use the Delphi method. To meet the purpose of the 
survey, conventional Delphi techniques were applied for forecasting the time of re- 
alization, i.e., the expected date of event occurrence. Modified Delphi techniques 
were used to evaluate the relative importance of individual subjects addressed. 
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6.1.3 Conduct of the Delphi Survey 
6. 1. 3. 1 Selection of Experts 

The number and type of experts selected for participation in the survey were 
guided by the following factors: 

1. Approximately 4, 000 experts were estimated for coverage of the various 
fields taking into account the expected recovery rate, i.e.,- based on past 
experience a number of experts contacted would not respond for one 
reason or another. 

2. Individuals with cultural as well as natural scientific background should 

be selected from the industrial, academic, government, and other sectors. 

3. Future-oriented individuals should be chosen. 

4. Some participants should be selected from persons having broad experi- 
ence and outlook with the expectation that their judgments would reflect 
a broad viewpoint. 

A preliminary inquiry was addressed to candidates, taking care to choose an . 
appropriate number in the various fields. Candidates were selected from a group 
made up of individuals recommended by government agencies, academic societies 
and associations, those individuals publishing forecasts in recent literature, and 
various persons in private organizations and information institutes expected to have 
specialized knowledge. In selecting the all-around group, preference was given to 
those individuals frequently mentioned by academic societies or listed in magazines. 
Respondents to the preliminary inquiry, the well established Japanese scientific and 
technical community, were requested to nominate members of the younger generation 
who were qualified for technological forecasting; this latter group was included as 
questionnaire addressees. 
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6. 1.3. 2 Setting Up Development Subjects 

The first questionnaire of the survey was a set of "hypotheses" to which the 
addressees were asked to respond by exploring subjects. The first questionnaire was 
sent to generalists and was intended as a step for identifying anticipated future socio- 
economic needs and exploring subjects in respect to these needs. Anticipating that 
needs in the next 10 years would likely be distinct from those 10 to 30 years from 
now, the forecast period was divided accordingly. The 1500 addressees were asked 
to evaluate the relative importance of given hypotheses as well as to nominate addi- 
tional development subjects, if possible, with respect thereto. Approximately 650 
subjects were set up by the five sectional groups on the basis of the results of the 
general study group's formulation of socioeconomic needs. The second and third 
questionnaires were for investigating the relative importance and the time of realiza- 
tion of the subjects set up by the foregoing procedure. These questionnaires were 
sent out to the original 1500 plus about 2500 specialists including researchers and 
engineers. Responses to the first questionnaire were sent out with the second 
questionnaire as reference data. The second questionnaire had columns for degree 
of specialization, evaluation of relative importance, time of realization, reason for 
nonrealization, and a comment. Since it is unreasonable to assume that all addres- 
sees would be familiar with all subjects, the specialization column was provided to 
determine the degree of specialization by the following code: 

1. Mark "h" (high) if you were formerly or are now engaged in research 
or job related to the subject. 

2. Mark "m" (medium) if you read books or other material related to the 
subject and have specialized knowledge, 

3. Mark "1" (low) if you only read about the matter in newspapers or general 
magazines or have heard about it but have not much specialized know- 
ledge. 
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The columns for inquiring about the importance of realization of the subject 
was contingent upon social, economic factors, and other conditions, assigning one of 
the following marks: 

1. "h" - very important 

2. "m" - important 

3. "1" - not important or unnecessary. 

The columns for time of realization were five-year periods up to the year 2000 
with two additional columns labelled "Unrealizable until 2000" and "Unpredictable." 

The columns for reason for nonrealization offered four choices, namely. Tech- 
nically Impossible, Social Restrictions, Economic Restrictions, and Other, in the 
third questionnaire these columns were replaced by a- number of choices entitled 
"Government measure," since the subjects were chosen from the government's 
standpoint. These latter columns solicited opinion about government policy to realize 
the particular development subject. 

The column labels were: 

1. Financing researches 

2. Training research personnel 

3. Coordinating researches 

4. Other. 

The meanings of these government measures are explained in the next para- 
graph. 

6. 1. 3. 3 Responses to Questionnaires 

About 74 percent of the 2414 addressees of the first questionnaire made valid 
responses. About 74 percent of the 4100 addressees of the second questionnaire made 
valid responses. About 78 percent of the 3108 addressees of the third questionnaire 
made valid responses. A tabular summary of responses to second and third 
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questionnaires is given in Table 6-1 for communication subjects, which were selected 
out of the total response in all areas. To the right of the subject column, the re- 
spective columns are explained as follows: 

1. Poll Number . The data for the second questionnaire is shown on top, and 
that for the third questionnaire on the bottom. 

2. Respondents . The total number of valid respondents for each subject is 
given. 

3. Importance . The distribution in percentage of the importance ratings by 
the valid respondents is given. 

4. Realizable/Unrealizable. The proportion (percent) of those forecasting 
realization by 2000 A.D. versus proportion (percent) of those forecasting 
nonrealization before 2000 A. D. 

5. Time of Realization. Graphic representation of inner quartile range of 
respondents on second poll by outline figure and on third poll by hash- 
marked figure; the peak of the figure indicates the median time. The 
distribution with regard to individuals with "h" specialization mark on 
the third poll are indicated by a straight line with a tiny circle; the circle 
marks the median and the extremities of the line the interquartile range. 

6. Reason for Nonrealization (Second Poll). The proportion of reasons for 
nonrealization in percentages of all respondents excluding those giving no 
specific realization time. The reasons are coded as follows: 

A: Technically impossible (unrealizable before 2000 A.D. , because of 

the many technical difficulties involved) 

B: Social restrictions (technically possible but unrealizable before 2000 

A.D. because of ethical, moral, social, or institutional restrictions) 

C: Economic restrictions (technically possible but unrealizable before 

2000 A.D. because the product, process, etc., cannot economically 
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Table 6-1. Communication Items in Japan's Science and 

Technology Survey 



6 































Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan's Science and 
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International data exchange will be carried out quickly as a result of direct access 
to overseas data banks. 

































Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 
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Programs will be fed directly to computers as Input; e. g. , flow charts will be 2 389 

fed directly to the computer without being encoded. 

3 314 



























Table 6-1. Communication Items in Japan’s Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan’s Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan’s Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 
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Table 6-1. Communication Items in Japan's Science and 
Technology Survey (Continued) 





































Table 6-1. Communication Items in Japan’s Science and 
Technology Survey (Continued) 
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compete with other similar processes, etc. , or because little market 
potential is seen) 

D: Other (unrealizable before 2000 A. D. for some other reason). 

7. Government Measures (Third Poll). The proportion of the most urgent 
government measures on science and technology in percentages of the total 
of valid respondents to a particular subject. The measures are coded as 
follows: 

A: Financing researches (sufficient funds including special tax conces- 

sions are needed to carry forward research and development in order 
to realize the development) 

B: Training research personnel (to realize the development, more 

research and development manpower must be supplied by training 
programs) 

C: Coordinating researches (to realize the development, there must be 

organic collaboration among research institutes and researchers and 
also, the organization for research and development should be im- 
proved) 

D: Other (some other measure is required) 

E No entry or "unnecessary" (those responses indicating the "Other" 
column and a comment to the effect that no government measure is 
necessary at all). 
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6. 2 TWO EXAMPLES CLARIFYING DELPHI TECHNIQUE 

In Systems Analysis and Policy Planning; Applications in Defense by E. S. 

Quade and W. I. Boucher, two examples of Delphi procedure are presented. The 
first example illustrates a simple application followed in seeking an answer to a 
narrow question, e. g. , estimating a noncomplex quantity. The second example 
describes the procedure when a much broader question is asked, e.g., selecting a 
policy. The two examples serve to clarify the principles of the Delphi technique. 

They are quoted in the following two paragraphs. 

6. 2. 1 Example 1. Choosing a Number by Delphi 

Consider the common situation of having to arrive at an answer to the question 
of how large a particular number N should be. (For example, N might be the estimated 
cost of a measure, or a value representing its overall benefit. ) We would then proceed 
as follows: First, we would ask each expert independently to give an estimate of N, 
and then arrange the responses in order of magnitude, and determine the quartiles, 

Q » M, Q , so that the four intervals formed on the N-line by these three points each 

X O . 

contained one quarter of the estimates. If we had 11 participants, the N-line might 
look like this: 



Second, we would communicate the values of Q , M, Q to each respondent, ask 

X O 

him to reconsider his previous estimate, and, if his estimate (old or revised) lies out- 
side the interquartile range (Q , Q ), to state briefly the reason why, in his opinion, 

X u 

the answer should be lower (or higher) than the 75 percent majority opinion expressed 
in the first round. Third, we would communicate the results of this second round 
(which as a rule will be less dispersed than the first) to the respondents in summary 
form, including the new quartiles and median. In addition, we would document the 
reasons that the experts gave in Round 2 for raising or lowering the values. (As 
collated and edited, these reasons would, of course, preserve the anonymity 
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of the respondents. ) We would then ask the experts to consider the new estimates and 
the arguments offered for them, giving them the weight they think they deserve, and in 
light of this new information, to revise their previous estimates. Again, if the revised 
estimate fell outside the second round' s interquartile range, we would ask the respon- 
dent to state briefly why he found unconvincing the argument that might have drawn his 
estimate toward the median. Finally, in a fourth round, we would submit both the 
quartiles of the third distribution of responses and the counterarguments elicited in 
Round 3 to the respondents, and encourage them to make one last revision of their 
estimates. The median of these Round 4 responses could then be taken as representing 
the group position as to what N should be. 

6.2.2 Example 2. Policy Advice from Delphi 

The Delphi technique can also be applied to broad policy problems. For example, 
let us consider how it might be used to uncover and evaluate measures that might help 
to speed recovery of a nation after a thermonuclear war. 

There are a number of reasons why an approach to this problem via the develop- 
ment of a mathematical model or a computer simulation might not be the most desir- 
able way to proceed. • If we had in mind six or eight fairly well defined and promising 
alternative postwar measures, we might consider adding a 'recovery' model to one of 
the many models that have been constructed to compute the damage caused by a nuclear 
attack. Assuming this could be done, the alternatives could then be compared in the 
traditional way used for comparing alternative force structures, employing a range of 
different war initiation scenarios and undertaking sensitivity analyses of the uncertain 
parameters. 

But the concept of 'recovery' is not very well defined. Very few of the many 
measures that might aid the survival of a nation or an area after a thermonuclear 
attack have been studied extensively. The emphasis so far has fallen primarily on 
measures such as shelters and active defense, which seek to reduce the immediate 
effects of the attack, rather than on measures to speed recovery after the initial effects 
of an attack have been experienced. Almost everyone has ideas about recovery 
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measures of this type that might be helpful, but seldom any well developed notion of 
their relative effectiveness and cost. Thus, there is a need to survey these ideas — 
to create an atmosphere in which they may be brought forth, subjected to critical 
review, modified and ordered according to various criteria with respect to their pos- 
sible effectiveness, acceptability, and costs, including social costs. The Delphi 
technique is well suited to this task. 

In addition to the presence of so many ill-defined alternatives, and the difficult- 
ies with the notion of recovery, there are a number of other reasons why an approach 
to the problem that puts emphasis on informed judgment is desirable. The decision- 
makers who would use the study would clearly be in the best position to judge the 
acceptability of measures that might either require radical changes in the prewar way 
of life or imply such changes for the postwar period — for example, how far to violate 
the rights of privacy or favor one sector of the economy or country over another if 
nuclear war were to come. But, their decisions would necessarily be based on many 
lowly but important relationships that require the intuition and judgment of specialists. 
Determining objectives — what we want to accomplish in the way of recovery and how 
we might distinguish one type of postwar world from another — must also be the res- 
ponsibility of the decisionmaker. But how to attain these objectives would require 
contributions from many disciplines. 

The alternative provided by the Delphi technique is to give up for the moment 
any attempt to compute the state of the postwar environment at various times after 
hostilities have ceased and instead to try simply to rank alternative prewar policies 
on the basis of the qualities that promise, in the judgment of specialists, to con- 
tribute the most to postwar recovery. This procedure cannot demonstrate beyond all 
reasonable doubt that a particular course of action is best. At most, it can assess 
some of the implications of choosing certain alternatives over others. But the system- 
atic searching out and partial ordering of promising steps could be extremely valuable. 

We should be under no illusion that for this problem a Delphi procedure would be 
the easiest thing in the world to carry out. In order to persuade the proper people to 
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authorize or to participate in such a study, the following points would have to be 
brought to their attention. One, the effort would not be intended as a substitute for 
other research. Two, if nothing else, it would highlight areas needing detailed study 
and in general, stimulate further work. Three, ideas provided in the course of the 
study — because of their possible half-baked character — would be kept anonymous unless 
attribution was specifically authorized. And four, the entire effort, in terms of man- 
power, could be kept quite minor, even though as much as 10 months might be needed to 
complete the study, since getting responses to questionnaires is just slow business. * 

Since the kind of survey being proposed is not a statistical survey of the Gallup 
type, but an attempt to generate ideas and to use the respondents to trace out the 
relationships among these ideas and the consequences of their adoption, it is im- 
material whether the respondents form a representative sample of the initially known 
points of view. What matters is that the viewpoints of persons with all major relevant 
backgrounds have a chance of being voiced. 

Assuming that our study would involve a range of experts both within and out- 
side the organization conducting it, the respondents might be organized into several 
"units, " so that the administrative task of running the experiment could be kept sim- 
ple. Each unit might consist of a central committee of three plus a panel of 6 to 12 
respondents. The committee chairman would be the person responsible for organiz- 
ing his unit’s activity, for maintaining liaison with the project director, and for trans- 
mitting the responses of his unit; One or more units might be located within the 
organization carrying out the study and the other units at some of the various places 

♦Incidentally, there exists an Act of Congress (5 U.S.C. Sec. 139, c-e [1942]) that 
forbids a government agency to conduct or to sponsor a study in which identically 
worded questionnaires are circulated to more than nine respondents without prior 
permission of the Bureau of the Budget. Since the intent of the Act is to keep 
businessmen from being bothered with a continuous stream of government forms — 
not to hamper scientific investigation — users of the Delphi technique whose support 
comes from government funds should not have difficulty obtaining such permission. 

Of course, one could confine the respondents (except for at most nine outsiders) to 
the research organization (this includes consultants) or the sponsoring agency. 
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where there is a concentration of respondents. Alternatively, the respondents might 
be dealt with directly or split into functional groups or disciplines such as ecology, 
economic growth, and so on. 

The inquiry itself could be broken down into four to six successive rounds, each 
based on a suitably formulated questionnaire. Only round one would necessarily 
involve all respondents. 

The first questionnaire would contain; in addition to the questions themselves, 
a brief background statement explaining the purpose of the study. It would include a 
statement that responses will be handled anonymously, except that approval for the 
use of names may eventually be asked in case certain suggestions are deemed worthy 
of being recommended for further action. Only the members of the steering com- 
mittee would initially be cognizant of the authorship of ideas. In the statement, 
suggestions would be included about keeping the proposals in practical operational 
terms and avoiding generalities. The respondents would be urged to include all 
suggestions that they think should be examined, even though they might be dubious 
about advocating them. 

The following sample questionnaire incorporates a number of these suggestions. 
Since it is addressed more to the readers of this book than to potential respondents, 
considerable reworking would be required before it could actually be used. 

Questionnaire 1 

This questionnaire is being submitted to you in an effort to elicit fresh ideas on 
what steps should be taken to reduce the problem of postattack recovery after a 
thermonuclear exchange. We are not looking for measures that reduce the manner of 
weapons impacting (ABM, for example) or measures that reduce their efficiency (such 
as shelters). Primarily we are looking for ways to help restore agriculture and 
manufacturing and the structure of society and government. An earlier study has 
suggested that the measures we are seeking to identify and weigh fall into three 
classes: preventive, which would aim at reducing the damage to our resources, such 
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as food stocks and water and power sources; emergency, which would attempt to deal 
with the distribution and management of supplies to sustain the population after the 
war; and long run, which would deal with recovery proper. Regardless of your feel- 
ings about the probability of nuclear war and the futility of such actions — in them- 
selves or in contrast to the results we might obtain if we contributed equal resources 
to deterrence — ask yourself what measures should be considered. 

This effort is being conducted very much in the spirit of a brainstorming ses- 
sion, except that it sets out to collect ideas in written form rather than through the 
give-and-take of open debate. At this stage, therefore, it would be entirely in order 
for you to submit ideas even if you yourself consider them half-baked, or if you 
merely regard them as worthy of further exploration without wishing to endorse them, 
or if they would only gain full meaning within an adequately elaborated context. 
Remember that this survey is in no way intended as a substitute for other research; 
indeed, its chief virtue might be to highlight areas needing detailed study and, in 
general, to stimulate further work. 

Question A. If you were a close advisor to the President, what actions would 
you advise him to consider taking (including recommendation of legislation to Congress) 
that might speed recovery after a thermonuclear attack? 

The following considerations — the list is by no means complete — seem rel- 
evant to this question. You may wish to delete or modify some items or add others. 
They are offered only to spark thought, and are listed randomly to avoid prejudging 
the order of importance or the feasibility of any measures. 

1. Since the control of infectious diseases could be a serious problem 
in the disrupted postattack environment, should current public 
health policies be reviewed for possible changes that would improve 
their effectiveness in a postattack situation? What policies? What 
changes? 
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2. A number of studies indicate that fires, both urban and wildland, as 
well as their sequelae of floods, erosion, and additional fire hazards, 
could be serious long-term problems in the postattack environment. 

Is there a need to review current fire prevention and control practices 
for possible changes and innovations that could improve our postattack 
capabilities to cope with these problems? What changes might be 
made? It has been suggested, for example, that we might undertake 
controlled burning prewar and also create appropriate firebreaks to 
prevent wildland fires from encroaching on contiguous urban areas 
or to keep urban fires from spreading to the countryside. We might 
also consider some steps to provide for reseeding burned areas post- 
attack to reduce erosion and flooding. 

3. How serious a problem would it be to find feasible alternatives for post- 
attack land uses that would be keyed to postattack requirements for 
food and other agricultural products? For example, what other crops 
could be grown on land too heavily contaminated with fallout to grow 
food, or what food crops could be grown on land not heavily con- 
taminated but now used to grow non-food products? 

4. What priorities should be observed in restoration of facilities post- 
attack? 

5. Should differential protection be provided for different segments of 
the population? 

6. Is organizational damage likely to be a serious problem in the post- 
attack environment? 

Question B. What research should be undertaken by the scientific and technical 
community that might either lead to or accelerate the discovery of measures that 
would help speed postwar recovery? 
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Again, here are a few possibilities that you may wish to consider in your res- 
ponse. 

1. Develop models. It might, for example, be important to build a 
flexible modular fallout model, or a model of the ignition and spread 
of urban fire and its impact on population in the fire area, including 
the protection afforded by available shelters against heat and carbon 
monoxide poisoning. A model of wildland fire that would relate igni- 
tion and spread to plant cover, season of year, weather, geographical 
region, and the nature of the nuclear attack might also be useful, as 
would models of a disrupted economy, since current models all seem 
to assume an organized society. 

2. Perform further research. Research in atmospheric physics, for 
example, might give us a way to estimate the effects of nuclear 
exchanges on weather and climate. Similarly, research might be 
undertaken on ecological disturbance or on the long-term genetic 
effects of radiation on man. (Both of these problems have already 
been studied in some detail, but much ignorance remains. ) 

3. Develop technologies for food storage and synthesis. 

4. Develop contingency plans for priorities in resource allocation by 
age, sector of the economy, or some other standard. 

Once the responses to this first questionnaire are received, the next, and 
hardest, step would be for the steering committee to sort and collate them, clarifying 
their meaning through checks with the respondents if necessary, eliminating obviously 
nonoperational suggestions, doing some minor editing and, hopefully, generating use- 
ful additions to the list. 

The list of proposals thus produced might then be submitted either directly to 
the original respondents or, as an intermediate step to obtain further refinement, to 
the "unit" committees. The result of this review might be the elimination of, say, 
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two-thirds of the proposals as being less promising. The remainder would then be 
annotated by the steering committee with brief arguments pro and con; they might also 
be ranked by merit according to some consensus formula. 

Because the wording of every questionnaire except the first depends on the outcome 
of preceding rounds, we can at best indicate only the form the remaining question- 
naires might take. The second might look something like this: 

Questionnaire 2 

The tabulation given below contains a list of tentative proposals to speed post- 
war recovery. We would like you to give us your judgment of each item in terms of 
its desirability, feasibility, and potential importance (assuming feasibility). 

For each item, check one box under Columns A, B, and C. In making this 
evaluation, consider the intrinsic rather than relative merits of the proposal. 


ABC 
DESIRABILITY FEASIBILITY IMPORTANCE 


NO PROPOSAL 

DESIRABLE 

MILDLY DESIRABLE 

DOUBTFUL 

MILDLY UNDESIRABLE 

UNDESIRABLE 

DEFINITELY FEASIBLE 

POSSIBLY FEASIBLE 

DOUBTFUL 

POSSIBLY INFEASIBLE 

DEFINITELY INFEASIBLE 

VERY IMPORTANT 

IMPORTANT 

SLIGHTLY IMPORTANT 

UNIMPORTANT 

1 ESTABLISH CONTINGENCY 
PLANS FOR PRIORITIES IN 
ALLOCATING RESOURCES 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2. MODIFY CURRENT PUBLIC 
HEALTH POLICIES TO IN- 
CREASE THE POSSIBILITY 
OF CONTROLLING INFEC- 
TIOUS DISEASES AFTER 
NUCLEAR ATTACK 


















k 


k 

k 



/ 


k 


□ 

k 
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This questionnaire would, of course, be accompanied by written arguments, pro 
and con, for each proposal listed. 

If the results of this appraisal indicate that an item ranks no higher than 'doubt- 
ful' in any category, it would be eliminated from further consideration. 

For the remaining items, some of which would obviously be controversial in one 
or more aspects, more exacting standards of acceptability would need to be set. The 
next questionnaire would explore the reasons for any divergence of opinions; it might 
take this form: 

Questionnaire 3 

The following items out of the list previously submitted to you have been elimi- 

i 

nated for the reasons checked: 


Reason for Elimination 

Item Description Undesirable Infeasible Unimportant 

1 X 

3 X 

4 X X 


The following items have been accepted as being desirable, feasible, and im- 
portant. 

Item Description 

11 

17 
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The remaining items are controversial in one or more respects. In those cases 
where a check mark is circled, your previously expressed opinion was at variance 
with the opinions of several of the other respondents. For each, please indicate very 
briefly why you hold this particular opinion. (For example, if, in Item 6, a check 
mark in the Desirability column is circled, please explain why you gave Item 6 the 
desirability rating you did in response to Questionnaire 2. ) Alternatively, if on recon- 
sideration you do not feel strongly enough about your previously expressed opinion to 
defend it, please indicate this by stating a revised rating. 


Reason 

Controversial as to Previous Rating 

Item Description Desirability Feasibility Importance or Revised Rating 

2 


5 


6 


If the replies to this questionnaire continue to move toward a consensus on some 
of the proposals, or if for some reason the apparently irreconcilable differences of 
opinion seem inadequately documented, one or more additional questionnaires may be 
worthwhile. In form, these would resemble Questionnaire 3. 

What might the final result tell us that we did not already know or could not ob- 
tain from less unconventional types of analysis? The answer can be very brief. Many 
aspects of the postattack recovery problem cannot be handled by standard cost effec- 
tiveness techniques. For example, how can one assess the effect on the arms race of 
a prewar measure such as the storage of materials for the recovery period? Our 
example suggests that the Delphi technique offers, at the very least, a way to approach 
such questions. 
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6. 3 UNIVERSITY OF MICHIGAN SEA GRANT INQUIRY 


6. 3. 1 Introduction 

The Sea Grant Program provided an excellent opportunity for critical evaluation 
of Delphi Techniques in an operational environment. The principal challenge was 
representing a wide range of opinion systematically in putting together the vital parts 
of long-range research and planning effort for water resource management. The 
associated Delphi exercises were designed to support Sea Grant goals of encouraging 
the involvement of university people in a concern for comprehensive management of 
the water resources of the Great Lakes, the integration of their informed judgments, 
and the communication of these judgments to citizens who are faced with decisions 
that will affect their region's social and economic development for many years to 
come. 

For the Sea Grant exercises, informed judgments were solicited in lieu of 
expert opinion. Consequently, a rationale existed for inclusion of politicians and 
concerned citizens on the panels and making the most of passing along pertinent 
information during the questioning process. Further, the concept of consensus was 
shifted from an emphasis on unanimity and a meeting of the minds to a measure of 
collective judgment. 

6. 3. 2 Overall Plan of Delphi Effort 

Three groups of panel members were formed for the Delphi exercise in the Sea 
Grant Program. The first group was technicians knowledgeable in water resource 
management. The second group was behaviorists, who in academic background and 
interests were oriented toward the social sciences. The third group was decision- 
makers, who were persons from the Grand Traverse Bay area believed to be influen- 
tial in that political process through which decisions on regional planning are made. 
The function of the three groups was considered to be reasonably consistent with the 
actual roles anticipated for such groups in deliberations and actions associated with 
planning for future regional development. 
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A progressive type of Delphi was planned, building from specific issues toward 
final group recommendations and research priorities. Previous Delphi exercises 
had been characterized by listing of technical events and requesting judgments thereon. 
Here; the Delphi effort was designed to identify potential technical, social, and politi- 
cal developments or to accomplish specific objectives. Special attention was given to 
the communication aspect of the method so that information could be most effectively 
transmitted to the decisionmakers; 

Broader-based panels were formed among, the three groups to assure inter- 
change of information useful in regional planning. 

In the phase dealing with potential technical, social, and political developments, 
each of three panels were planned to have four rounds of questionnaires. The first 
round asked panel members for suggestions and estimates regarding future events 
affecting marine resources in the Grand Traverse Bay area in the next 20 years; 
also, they were to evaluate several items according to statements on an evaluation 
matrix in order to familiarize themselves with numerical estimates that would be 
requested on subsequent rounds. On round two the discrete items identified in round 
one plus some additions were to be considered. Self-evaluation estimates were to be 
requested. For each item estimates were made of relative importance, economic 
and technical feasibility, and probability and timing associated with developments 
with which panelists had some familiarity. Further, on this round, panel members 
were to suggest related developments and requisite technology. On the third round 
items were screened and those that were unimportant,, that lacked panel member 
insight thereon, or that had ambiguous specification were to be dropped from further 
consideration. Progressive feedback of information; including statistical summaries 
were to be provided from previous rounds. In the final round the technicians were to 
make additional probability estimates for pairs of events that panel members had 
suggested were closely related, considering whether or. not occurrence or nonoccur- 
rence of one event would affect the outcome of a following event. 
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In order to provide a point of reference in making other subjective judgments 
panel members were to project trend curves of statistical values that ordinarily are 
used to measure socioeconomic growth of a region. The technical panel would be 
requested to produce supporting studies on pollution sources and recommendations 
regarding waste water treatment and disposal systems for use by broader-based 
panels. 

6. 3. 3 Methodological Modifications 

In carrying out the Sea Grant Delphi exercise, certain significant modifications 
and refinements in methodology were developed. In particular, panelists indicated 
that emphasis should be given to: 

• Safeguards against a manipulated consensus 

• Scaling of words and phrases 

• Aids to personal probability assessments 

• Relationships among forecasted developments. 

Safeguards were taken by coding each suggestion by panel member number and 
associating basic biographical information, by retaining all pertinent comments with 
minimum editing, by carrying over estimates from a previous round if respondent 
missed a round, by full information feedback, and by specific request of supporting 
arguments from those respondents whose estimates for the previous round were 
outside of the consensus range. (To the extent that other panelists were influenced 
by these arguments, the consensus would tend to broaden rather than narrow. ) 
Unfortunately, the biographical data compromised the anonymity of some panel 
members. On the other hand biographical data gave weight to suggestions and com- 
ments and assured panelists that they were in a peer group. 

The scaling of words and phrases were considered critical in the Delphi method 
for obtaining and communicating subjective judgments. It was found that panelists 
had wide variance in their viewpoints of verbal phrases corresponding to commonly 
used numerical probabilities. In the opening round of the Sea Grant Delphi, the 
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participants were asked to make such associations. The substantive results were 
that many words are inappropriate for use with the Delphi method and that excessively 
refined probability estimates cannot be achieved due to inconsistencies of meaning 
among panelists. Among all the words or phrases considered, those corresponding 
closely to the commonly used probabilities of 10, 25, 50, 75, and 90 percent are 
shown in Table 6-2. A Delphi procedure was used to determine whether feedback and 
reassessment would result in greater agreement on numerical probabilities associated 
with several expressions. It was found that the dispersion of estimates narrowed 
while only slightly changing the measures of central tendency. 

A guide for making personal estimates of probability was sent to all members of 
the broad panels. It was believed that many panelists had little experience with 
assessment and interpretation of personal probabilities; it was believed they would 
have problems in making consistent assessments. The results of the research on 
scaling words and phrases gave strong empirical evidence that there are specific 
words and phrases denoting the likelihood of a future event which multidisciplinary 
groups associate with numerical values closely corresponding to the probabilities of 
10, 25, 50, 75, and 90 percent. A remarkable degree of agreement was obtained 
between Sea Grant researchers and decisionmakers. 

In investigating the relationships among forecasted developments, the Sea Grant 
administrator used the expertise of the panel members and inherent Delphi techniques 
in lieu of cross-impact analysis. Several procedures were employed to encourage 
respondents to assess intuitively the relationships among future developments under 
considerations: 

• Correlation of different points of view of multidisciplinary groups. 

• Preview by technical experts of issues particularly vulnerable to 
technological advances. 

• Identification by panelists of closely related developments. 

• Concurrent assessment of related issues and developments. 
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Table 6 - 2 . Words or Phrases Corresponding Closely to 
Commonly Used Numerical Probabilities 
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• Request for conditional probability estimates by panelist of "paired" 
developments. 

Most panelists altered their final estimates on those developments subjected to 
conditional probability assessments, indicating that relationships had not previously 
been fully considered. With the strong emphasis on relationship among events through- 
out the Delphi exercise, any movement in the final estimates as a result of the con- 
sideration of specific conditioning effects was believed to be significant. One of these 
groups of developments and the evaluation matrix is shown in Figure 6-1. The events 
for this particular series of interrogations will be referred to either as dependent 
events or as conditioning events. Any single event may be placed in either category 
as the interactions among pairs of events are sequentially considered. 

A sample of an individual response is shown in Table 6-3. The summary 
includes the respondent's evaluation of his competence to judge the development — using 
a scale ranging from 1 (unfamiliar) to 5 (expert or researcher in the area), his initial 
estimate of the probability of the development occurring in 1971-80, his third round 
estimate, his final estimate, and his conditional estimates. The technique of system- 
atically obtaining and analyzing conditional probability estimates proved particularly 
valuable for: 

• Encouraging respondents to reexamine their estimates in view of related 
developments. 

• Assessing extent of panelists' intuitive relationship of anticipated develop- 
ments. 

• Identifying the influence of conditioning events and the uncertainty associated 
with their effect. 

• Assessing the impact of alternative strategies that will most effectively 
influence the outcome of important developments. 

• Providing a mechanism, in combination with self-evaluation and confidence- 
indexes, for weighing individual estimates. 
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Panel Member 12 Panel T-l 
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Figure 6-1. Interrelated Developments, Group 3 



Table 6-3. Conditional Probability Estimates of Individual Respondent 
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♦Computed probability of the dependent event using the formula P(A) = P(B)P(A/B) + P(B)P(A/B). 



• Determining some of the respondents' assumptions regarding overall social, 
political, and technical environments when making specific judgments. 

6.3.4 Evaluation of the Sea Grant Delphi Exercise 

The conduct of this exercise went closely according to plan. The panelists were 
interested in improving Delphi techniques to integrate the judgments of a multi- 
disciplinary research team and to convey informed insights to society. In the second 
round of questionnaires the panelists were asked to provide estimates on 75 discrete 
items identified in round one. The several estimates on each item compounded the 
work effort of the panelists to the point where they complained that estimates took too 
much time and became tedious. In the third round a special questionnaire form was 
sent to technicians soliciting their views on the positive and negative aspects of Delphi 
and appropriate areas for further application. In fact, this was a Delphi procedure. 

The results indicated that the Sea Grant Delphi exercise corresponded rather closely 
to the panelists concept of an ideal treatment. 

Additional evaluations showed that among the participants the techniques were 
more highly regarded by groups formed on the basis of broad ranges in training and 
experience than by technicians — the group most administrators of the Delphi technique 
have focused on. Evaluation results also supported tailoring the method to groups on 
the basis of their background, training, and experience. 

6. 4 TRW PROBE II DELPHI STUDY 

Modified versions of the Delphi techniques, called Probe I and Probe II, were 
developed by Harper A. North and Donald L. Pyke for internal use by the Thompson- 
Ramo-Wooldridge Corporation (TRW). This proprietary program has been discussed 
in various aspects in at least three different articles and Bright and Agers books on 
technological forecasting - all of which are cited in the bibliography. TRW has applied 
these Delphi approaches to collect, from its large and diverse staff, new ideas for 
products and product areas. The Delphi procedure has proved useful as a means of 
communication among division technologists, division executives, and corporate 
managers; it has served to integrate corporate goals and objectives with technologically 
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feasible capabilities. The Delphi supplemented trend extrapolation in TRW's tech- 
nological forecasting to aid R&D planning. Such a technically based company must 
examine the face of the existing technological explosion and estimate the pace with 
which their industry must cope during the years ahead. For TRW purposes long- 
range forecasting was defined as the prediction of likely inventions, specific scientific 
refinements, or discoveries in technology, including new applications, or products 
which may become possible. Scope of forecasts were limited to events likely to have 
a more or less direct impact on the company. TRW approaches technological fore- 
casts, because all these forecasts serve the same final purpose - namely, to help 
management improve its decisionmaking processes by providing immediate information 
on probable and significant future trends. 

6. 4. 1 Probe I 

TRW's first Delphi study, called Probe I, was completed in June 1966. The 

basic approach taken was patterned after Helmer and Gordon's, "Report on a Long- 

( 2 ) 

Range Forecasting Study. " Generally, this initial effort was a test of the feasibility 
of using a Delphi method in an industrial environment. Twenty-seven senior technical 
experts from different divisions and departments were polled directly, individually 
and anonymously. They were asked to anticipate major technical "events" occurring 
in the period 1966 to 1985 that would have major potential impact on TRW. A list of 
401 events, sorted into 15 categories were compiled. These items were more the 
extrapolation of the current technology rather than speculative long-range events. 

A review of R&D activities during the latter half of 1966 revealed that Probe I had 
been a useful checklist and a vision-extending exercise for the panelists and others, 
but that no real evidence existed that it had been used directly in R&D planning. The 
principal benefit the company received was a great deal of publicity concerning its 
foresight. However, the results of Probe I were sufficiently provocative to indicate 
both significant threats to one part of TRW's business as well as major opportunities 
to other, groups. It was decided, therefore, to undertake a major in-house study of 
these areas, adding a rather elaborate second stage to the original survey. 
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6. 4. 2 Probe IT 


Probe n was launched late in 1967 and completed by December 1970. The 15 
categories of events determined in Probe I were refined to focus more closely on the 
current interests. Probe II followed a revised procedure taking advantages of lessons 
learned in Probe I. 

1. The second version assumed a socioeconomic environment, essentially 
TRW's 1975 long-range corporate business plan, whereas, in Probe I 
panelists made own assumptions. 

2. Subject matter was restricted to 15 specific areas. 

3. The number of participants was expanded. Each operating division was 
represented if possible by an expert in each of 15 categories in which the 
division held an interest. This led to the selection of 140 experts. Figure 
6-2 lists the categories and indicates the applicable divisions interest as 
well as presence of expert opinion in those divisions. 

4. More penetrating questions concerning events were sought in lieu of the 
limited inquiry in Probe I asking for only the most probable date of 
occurrence. 

In round one of the questioning each panel member was asked to list probable 
technical events by category and make three separable evaluations as follows: 

• Desirability (needed desparately; desirable; undesirable but possible). 

• Feasibility (highly feasible; likely; unlikely but possible). 

• Timing (year by which probability of occurrence exceeds 10 percent: 

50 percent; 90 percent). 

Desirability was to be considered from the viewpoint of the customer, i. e. , as 

reflecting an estimate of demand. 

Feasibility was to be considered from the viewpoint of the producer, reflecting 

both technical possibility and difficulty in development. 
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Key "X" indicates the presence of expert opinion in a division for each category. 


Figure 6-2. Probe II Categories and Divisions' Interests 










































Timing was to reflect an estimate of the date by which there is reasonable 
chance that the event may have occurred (p = .1), the date by which the event 
is almost certain to have occurred (p = . 9), and the estimated date by which 
there is a 50/50 chance that the event will have occurred (p = . 5). 

Round one produced 2100 predictions. Four critical editings, to eliminate dupli- 
cate and trivial or irrelevant items, reduced the list to 1438. In the course of editing, 
statements were modified to improve clarity and avoid distortions of intent and 
meaning. 

In the second round each panel member received a composite list of the edited 
predictions contributed by his panel plus those from other panels related to his cate- 
gory. The panelists were asked to evaluate all events with respect to the same three 
factors considered in round one. Estimates made by the originator of an item were 
not included; each panelist was on his own. Each expert was specifically asked to 
make sure that his evaluations and estimates were consistent with each other, 
especially with regard to expected date of occurrence. Printing of round two question- 
naires and processing of the responses was accomplished by computer. The computer 
was an essential tool because the 1438 events considered by varying numbers of the 
140 panelists involved 22, 000 judgments, each containing seven bits of information. 
Moreover, the subsequent sampling of data by individuals seeking specific information 
would be exceedingly difficult without computerization. Figure 6-3 shows a sample 
question from a round two questionnaire with hand-written responses. Note that each 
panel member is assigned a code number (here, 304) by which he may be identified, 
if necessary. 

The number 201030 references the event statement to a particular source 
document. The numbers 01, 02, 04, and 09 indicate that this event was evaluated by 
the panels on Electronics, Materials, Power-conditioning, and Transportation. The 
responses indicate that the evaluator considers that: 

• He is a specialist in a number of relevant technologies 

• The occurrence of the event will be demanded by a significant segment of 
the public 
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PROBE CATEGORY: +9 


TRANSPORTATION 


PANEL MEMBER 304 


EVENT 

NUMBER EVENT DESCRIPTION 

PANELS 

EVALUATING 

FAMILIARITY 

DESIRABILITY 

FEASIBILITY 

PROBABILITY 
OF EVENT 

DATES 

201030 ELECTRIC AUTOMOBILES 
USING FUEL CELL POWER 
OR FUEL CELL/BATTERY 
COMBINATION WILL BE 
MARKETED COMMERCIALLY. 

01.02.04.09 

1. FAIR 

2. GOOD 

3. EXCELLENT 

1. NEEDED 

2. DESIRABLE 

3. UNDESIRABLE 

1. SIMPLE 
"2. POSSIBLE 
3. UNLIKELY 

0.7 

1. '79 

2. '85 

3. '90 


Figure 6-3. Sample of a Round Two Questionnaire 


• Substantial technical effort will be required 

• The probability is . 7 that the event will occur 


• Assuming that the event will occur, he expects it to occur during the 
period between 1979 (. 1 date) and 1990 (. 9 date), with the most probable 
date (. 5 date) being 1985. 

In the informal third round TRW resolved any wide differences of opinion con- 
cerning events with estimated dates of occurrences falling outside of the median range 
(. 1-. 9) by discussing the predictions individually with the panelist involved. At the 
end of this round each event, on a statistical average, was evaluated by approximately 
40 experts representing three or more different panels. Figure 6-4 shows how the 
event of Figure 6-3 may appear in its final published form (with a fictitious evaluation). 
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ELECTRIC AUTOMOBILES 
USING FUEL CELL POWER 
OR FUEL CELL/BATTERY 
COMBINATION WILL BE 
MARKETED COMMERCIALLY 

+.7 

+.4 

.8 



— 
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Figure 6-4. Sample of Event in Final Published Form 
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The composite evaluation by all panelists questioned on the electric automobile 
indicate that: 

1. On a scale ranging from +1. 0 (desirable) to -1. 0 (undesirable), the 
numerical average of evaluation of desirability is +0. 7. 

2. On a scale ranging from +1. 0 (simple) to -1. 0 (unlikely), the numerical 
average of the evaluations of feasibility is +0.4. 

3. The numerical average of estimates of the probability that the event will 
occur sometime is 0. 8. 

4. If the event does occur, the most likely date for it to happen is 1984 
(the median 0. 5 date); the period of expectancy range from 1977 (the 
median 0. 1 date) to 1987 (the median 0. 9 date). 

Hindsight suggests still other improvements to Probe II. TRW assumed, and 
with good reason, that engineers can forecast markets for radically new products. 
This assumption was based on the belief (still held) that the vendor's engineers and 
the customer's engineers make a better working team for speculating about the need 
for radically new products than do the vendor's salesmen and the customer's buyers. 
TRW plans in future Probes to include perceptive market managers and even soci- 
ologists to introduce more maiket-oriented factors and thus provide a better over- 
view of the environment of the future. Another possible variation, which has been 
considered, is the submission of questions from operating managers to the Probe 
computerized data bank and, in addition, to the current panelists via the existing 

i ■ ' 

computer program. 
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SECTION 7 - UTILIZATION OF DELPHI-OBTAINED DATA 


X 8 

Harold A. Linstone states that forecasting is only one of the first steps in 
planning; what one does with a forecast is important, and planning is what one does 
with it. After outlining major weaknesses of present-day forecasts, Linstone 
provides a checklist of what constitutes a good forecast, as follows: 

• Does it provide useful planning insights? 

• Does it combine extrapolative and goal- oriented methods in a feedback 
mode? 

• Does it deal specifically with uncertainty? 

• Does it deal with the total system? (In lieu of parameters. ) 

• How holistic an image does it present? (i. e. , an image presented so well 
that a manager can place himself in the situation and find it meaningful 

in a way that can never be achieved with numbers and curves alone. ) 

• Can the forecast be replicated by other means? (i. e. , can the forecast 
be reproduced using different techniques and personnel?) 

• To what extent can the forecast be validated? 

• How suitable is the forecasting team? (i. e. , as diverse a group of thinkers 
as possible. ) 

Identification of telecommunications needs and priorities alone is not enough; 
the likelihood of occurrence of these developments as a function of time must also be 
determined (customary in Delphi studies). It must be recognized that the likelihood of 
occurrence of any one event typically depends upon the occurrence or nonoccurrence 
of other future events which precede the event in question (see Paragraph 5. 8 and 
Appendix A on cross-impact and contingency analyses). A determination of needs is 
a normative approach to forecasting rather than exploratory. The normative approach 
focuses on future goals and values (needs). One must then move from the future to the 
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present and ask how do we get there. This latter question is exploratory. Hence, for 
a forecast to be meaningful, it is vital that the normative and exploratory approaches 
be linked. 

A Delphi statement of long-term communications needs and priorities is not an 
absolute, definitive assertion of what the future will be, but rather a probabilistic 
statement about the future with expressed confidence level. This may be useful to 
NASA in setting realistic and correct organizational goals. Also, such a forecast 
serves NASA management by stimulating awareness of what the future may hold in 
store sufficiently early for appropriate action to be initiated. Of course, to a very 
marked degree, the future will be what we (the present generation) make it. One 
cannot describe all the possible futures. It must be borne in mind that whatever 
statement of communications needs and priorities is made does not constitute that 
exhaustive enumeration of possible "states of the world" which is called for by 
decision theory. A decision theory model, although neat and intellectually elegant, 
may be a useful first approximation, but may be misleading in that the chances are 
very great that the "state of the world" that does materialize will be one which was 
not perceived in advance. Further, both the optimal strategy and the strategy finally 
chosen are likely to be different from those which were arrayed in the payoff matrix. 

It is presumed that a NASA R and D manager will face a decision on the Delphi 
statement of needs and be held accountable for end results (i. e. , his guidance and 
definition of a path of progress toward fixed goals). In the chain of decisionmaking, 
experts must appear somewhere, distinct from the Delphi expert inputs on communi- 
cations needs and priorities; there is no substitute for expert knowledge of the / 
subject matter. Before initiating a NASA Delphi study of telecommunications needs 
and priorities, it would be well to anticipate the decision process at the end and 
include questions leading to some quantifiable measures of decision criteria 
(scientific values) by some subpanel of qualified experts or a separate panel of policy- 
making experts as used in the Sea Grant Delphi Study (see Paragraph 6. 3). It is not 
necessary nor recommended that the decision itself be delegated to outsiders, whose 


7-2 



professional interests may be partially involved and therefore susceptible to bias. If 
Delphi study results are simply circulated without specific advance measures for 
incorporation in the planning cycle, it may turn out that the Delphi report is viewed 
with idle curiosity or as a checklist, very much like what happened after TRW's Probe 
I Study (see Paragraph 6. 4). 

The data obtained from Delphi methods may be insufficient to make a rational 

decision concerning future events. Most decisions are made with some model in mind 

but are usually combinations of qualitative and quantitative judgments. A hybrid 

decisionmaking system attack on problems is considered superior to others since it 

brings into play intuitive and analytical approaches which support each other. Such a 
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system is shown below in Figure 7-1. 

INFORMATION 


QUALITATIVE JUDGMENT QUANTITATIVE JUDGMENT 



The flag shown occurs when the two approaches disagree. By careful reexami- 
nation of intuition and analytical methodology, better forecasting can be accomplished 
for effective decisionmaking. 

Since an unlimited amount of money is not available to develop all telecommuni- 
cations systems, we can conceive in the real world decisionmaking process attempts 
to evolve optimum cost-effective mixes of communications systems with limited 
resources of future dollars. Therefore, the Delphi questions (see Paragraph 4.2) 
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should be formulated not only to obtain meaningful responses on telecommunication 
needs and priorities but also to provide: (1) probability of events occurring, or 
weighing of importance of a criterion; (2) criteria for decisions; and (3) utility values 
for different communications systems. These will enable the decisionmaker to make 
objective decisions on allocation of funds to mixes of communication systems. 

If classical decision theory is to be used, the final answers obtained from the 
Delphi study must be appropriately formulated to fit such process. A short disserta- 
tion on classical decisionmaking follows. Again, it is emphasized that this technique 
provides only a first step in a decision process dealing with forecasts. 

7. 1 DECISION MATRICES AND CRITERIA 


With a variety of questions and iterated consensus of opinions, the Delphi 
analyst should produce an output which compares various communication systems. 
This must be presented as a table or matrix showing the criteria associated with each 
system. These criteria might include: 


Grade of service 
Reliability 
Travel 
Avoidance 

Information retrieval 
Life cycle costs 


National prestige 

Demographic migration projections 

Economy of reaching remote 
locations 

Flexibility for military and indus- 
trial reorganizations 

State-of-the-art track records of 
performance 


In general, the techniques should not frame the decision but should rather aid in 
making a decision. From here, results are given to the decisionmaker, who 
combines the inputs from other system analysts (who may be concerned with quite 
independent and/or exclusive problems) in order to select the "best" communication 
system mix. 


The art of choosing and applying appropriate criteria is very difficult. The 
decisionmaker must review the criteria to be applied to the systems inherent or 


7-4 



implied in the statement of long-term communications need and priorities presented 
by the Delphi analyst and actively review each system development vis-a-vis the 
criteria. This is not necessarily a simpile serial process. To the contrary, the 
decisionmaker may determine that additional information from the planning analysts 
is required and may define the bounds of the investigations required for sufficient 
definition of further communications needs. Upon completion of the study, the output 
is presented to the decisionmaker. This process may require several iterations. 

It must not be presumed that, with the establishment of "good" criteria, the 
decisionmaker simply selects the communication system development that most closely 
satisfies the criteria and implements his decision. The process of decisionmaking is 
shown below: 

1. A set of alternative actions on communication systems the decisionmaker 
might select. 

2. A set of conditions which reflect the possible environment or criteria 
desired in which the decision is to be made, often called states-of-nature 
of states- of-th e-world. Some criteria have been mentioned above. 

3. A set of outcomes which may result depending upon which action is chosen 
and which of the environmental conditions exist at the time the action is 
taken. 

4. A value or utility to the decisionmaker resulting from the outcome. 

5. Some assessment of the likelihood or probability of each of the states of 
nature being the true one when the action is taken. When we evaluate com- 
munication systems, weighting factors may be used in place of probabilities. 

Professor Ward Edwards of the University of Michigan, a noted decision-theorist, 
aptly described the above as follows: 

"All decisions can be analyzed by means of a rather small set of concepts. 

They are: 
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1. The acts . These are different things the decisionmaker can do, defined 
so as to be mutually exclusive. 

2. The states . The outcome of an act depends not only on its identity but also 
on factors not under the direct control of the decisionmaker. Such factors 
are called states, short for states- of-the-world, and are defined so as to 
be mutually exclusive and exhaustive. A probability is always associated 
with each possible state; sets of states vary in the precision with which the 
probability distributions over them can be ascertained. (A probability is 
your orderly opinion about how likely an event is, and so describes you 
more than it does the event. "Orderly" simply means that the sum of the 
probabilities of any exhaustive set of mutually exclusive events must be 

1. 0 - a very demanding requirement. Most people do not have such 
orderly opinions, a fact with serious consequences for decisionmaking. ) 

3. The outcomes . Each combination of an act and a state is called an out- 
come. Associated with each outcome is a payoff. Whether or not the payoff 
is defined numerically, it is convenient to assume that it has a subjective 
value or utility to the decisionmaker. Such utilities may sometimes be 
calculated, must more often be judged. " 

7. 2 EXAMPLE OF DECISIONMAKING DISCIPLINE AND UTILITY VALUE 

A hypothetical illustration of how classical decisionmaking could be used for 
deciding the proper allocation of funds to new telecommunications systems is presented 
in Figure 7-2. 

The alternatives chosen were feasible telecommunication means. The criteria 
are the columns or parameters against which each of the telecommunication systems 
is evaluated. The numbers in the matrix are obtained by using a Delphi consensus 
and a utility scale such as that illustrated. The weights or importance of each 
criterion may also be obtained by Delphi methodology using a scale of 0 to 100 as 
mentioned. 
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Criteria 



Communications 

Objectives 

Volume and 
Loading Require. 

Quality and 
Grade of Service 

Operating 

Requirements 

Growth 

Potential 

Cost 

bo 

c 

Xi 1 
CJ 03 
Cl o 
CD O 

« 

O -u 

» C « 

to C c 
ca cu o 
H K S 

Satellites 

7 

10 

8 

9 

8 

6 

10 

High Frequency Radio 

3 

2 

4 

9 

2 

10 

9 

Microwave 

9 

8 

9 

7 

9 

5 

6 

Tropospheric Scatter 

5 

5 

6 

8- 

3 

wm 

8 

Cable 

9 

9 

8 

8 

ffjfjfj 

H 

7 

Weight or 
Importance (0-100) 

100 

100 

90 

70 

100 

100 

80 


Figure 7-2. Decision Matrix Hypothetical Telecommunication System 

A general decision model is discussed in Paragraph 5. 4 and a general matrix 
shown in Figure 5-2. 

7. 2. 1 Utility-Value 

As has been shown, the utility or value is extremely important in providing 
objective decisions. For example, how do we go about putting in a utility or value on 
some measurement scale which exhibits the goodness orbenefit of using a satellite to 
communicate news, educational materials, etc. , to remote regions (e. g. , Appalachia) 
as compared to a single telephone at the local country store? 

Without going into the historical background of utility theory, well covered in 
texts by Edwards, Fishbum, Savage, and Stigler, a discussion- of the types and appli- 
cation of utility in decision models may be useful. 
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7.2.2 Types of Utility 


The concept of utility may be viewed as a measure of a pleasure-pain continuum. 
Pleasure is represented by positive utility and pain by negative utility. The decision 
or course of action yielding the most pleasure or utility is the obvious choice of the 
decisionmaker. Utility can be, and often is, measured in terms of dollars. Regard- 
less of the units in which it is measured, there are two fundamental types of utility: 

1. Ordinal utility 

2. Cardinal utility. 

7. 2. 2.1 Ordinal Utility 

Ordinal utility involves the ranking of alternatives in order of the pleasure or 
utility associated with them without specifically assigning magnitudes. This might be 
accomplished by assigning utilities such as excellent, good, fair, and poor. 

The only necessity is to establish a relative ranking. If utilities are assigned 
to alternatives A, B, C, . . . , n, a choice can be made as long as the ordinal scale of 
utility establishes that, for example, the utility of A is greater than B, B greater than 
C, etc. 

U A> U B> U C> •" "n 
where: u^ = utility of alternative A 
iLg = utility of alternative B 
u_ = utility of alternative C 
u^ = utility of the nth alternative 

Ordinalists contend that measurement of subjective utility on an absolute scale cannot 
be done meaningfully. However, it is believed that a preference between states in 
terms of relative utility can be established. 
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The use of indifference maps to assist the decisionmaker is a basic tool in 
applying the ordinal theory of utility. A typical indifference map is shown in 
Figure 7-3. 



The quantities of two different states-of-affairs or commodities A and B are 
plotted. The preference for particular quantities of A and B are considered in terms 
of the ordinal utilities corresponding to various combinations of A and B. No prefer- 
ence exists between the combinations of A and B that fall on a given curve. For 

example, there is no preference between the points A B , and A , B on curve u 

11 2 2 1 

Curve u 2 represents a higher ranking utility curve, u^ still higher, etc. It is signifi- 
cant, however, that curves on an indifference map need not always be in order of 
increasing utility. 


7. 2. 2. 2 Cardinal Utility 


In constructing cardinal utility tables, it is usual to determine as a starting 
point a state-of-no-pleasure-or-pain and assign to it the value of zero utility. A 
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given state-of- affairs or a unit of a commodity relevant to the previous starting point 
is assigned a value of one "util. " The unit of utility measurement is frequently the 
util. A comparison of alternative states of affairs or commodities within the frame- 
work of the two reference points defined is then made and appropriate values of utility 
assigned. Cardinal utility maps can be constructed by using the above approach. 


Quite often the concept of marginal utility must be considered. If one unit of a 
commodity has a value of one util, it is unlikely that two units of the same commodity 
will have a combined value of two utils. The second unit may have a marginal utility 

of only 0. 9 utils while still a third might have a value of only 0. 8, etc. Therefore, 

1 

utility can be considered as a function of the quantities of several commodities 
C (i = 1, 2, 3, .... n) and may be represented by a utility function 


U = f(C , C .... C ). 
12 n 


The marginal utility for an increase in ith state would be the partial derivative of U 
with respect to C^. The decisionmaker should choose from several alternatives that 
one which yields the highest increment in utility. 


The probability of a particular outcome or future state occurring given that a 
certain alternative or course of action is chosen must also be considered. The value 
of utility assigned a particular outcome must be modified by the probability that the 
outcome will occur. Utility multiplied by the chance of accomplishment should be the 
basis for effective decisionmaking. 


7. 2. 2. 3 Dollars as a Measure of Utility 

For a given decision problem, all outcomes must be measured on one scale of 
utility so we can compare them. The value of an outcome, or the measure of its 
utility, is the result of a subjective evaluation by the decisionmaker. An obvious 
scale for value is dollars, but this is often an inappropriate measure. Consider, for 
example, the value of lives lost. 

These values can be measured on a scale related to dollars: cost-effectiveness. 

It is often said that utility is maximized when the ratio of cost ($) to effectiveness (lives 
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saved or targets damaged) is minimized. Here again, there are cases where effective- 
ness is a subjective measure, such as pleasure for the cost of entertainment, or the 
peace-of-mind from insurance, or excitement from a lottery ticket. 

The reason dollars cannot be generally used for utility is that the subjective 
value of money is not linearly related to dollars. For example, consider a game 
where the chances of "win" or "lose" are equally likely, and the payoffs are thus: 

$ Win (P = 1/2) $ Lose (P = 1/2) 


a. 

1.50 

1. 00 

b. 

15. 00 

10. 00 

c. 

150. 00 

100. 00 

d. 

1, 500. 00 

1, 000. 00 

e. 

15, 000. 00 

10, 000. 00 

f. 

etc. 

etc. 


At what point on this scale would the bet no longer look attractive? Many people would 
say at level b. or c. , or perhaps d. Yet the ratio of win to lose remains constant and 
favorable. This type of reaction can be generalized as shown in Figure 7-4. 



Figure 7-4. Relative Attractiveness of a Bet 
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7. 3 MATHEMATICAL BASIS FOR DECISION PROCESSES 


7. 3. 1 Decision Rules 

At this point, one may ask "Given the matrix of utility values and weights of 
Figure 7-2, how do I pick a unique telecommunication system or combination of them 
to yield an optimum return for my money? " A decision rule is used to state how we 
will make our choice. Some of the more common decision rules are now shown. 

7.3.2 The Minimax (Maximin) Rule 

This rule is borrowed from game theory in which nature as a nonmalevolent 
opponent is the other player in a two-person zero-sum game. It is sometimes called 
(he "pessimist's rule" or the rule of the conservative person who wants to protect 
himself against a high loss or one who is willing to settle for a minimum gain. It is 

i 

normally called the minimax rule because many decision problems are structured in 
terms of loss tables. 

The rule states that, if our table represents losses, we should find the largest 
loss for each action and then choose that action with the minimum of the maximums 
(thus the name, minimax). Thus we are assured of losing the least in the worst case, 
and we might do better. Hence, it is a "worst-case" or conservative rule. This may 
be expressed mathematically as 

Min Max u.. 


which may be interpreted as for each action a., select the largest utility u.., then 

choose that action a. which has the smallest of the selected utilities. 
i 

In the case of the example in Figure 7-2, we want to select maximin or 


Max 

Min 

u.. 

i 

j 

V 
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Looking at the values in the matrix, we would have the following minimum values: 

Maxim in 



and would therefore select action a^ (satellite) which has the maximum of the minimum 
values. This assures us of never getting less than 6 utils. 


7. 3. 3 The Maximax Rule 


Corresponding to the pessimist or conservative person, there is always the 
optimist or gambler. This rule says choose: 


Max Max u.. 


or simply find the largest utility in the matrix (assuming gains) and choose the corre- 
sponding action. 

7.3.4 The Hurwicz Pessimism-Optimism Index 

This rule is attributed to Prof. Leonid Hurwicz, a noted statistician and decision- 
theorist. Prof. Hurwicz reasoned that most people were neither ultraconservative nor 
gamblers but rather were some place in between. Combining the minimax and 
maximax criteria, he developed the following rule: 

Choose that action which gives 


Max ( a Max u.. + (1- o) Min u..) 

ij . ij 

i J J 
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where a is called the pessimism-optimism index of the decisionmaker. It can be seen 
that this reduces to the minimax rule when a - 1. 

7. 3. 5 The La Place Criterion 

The La Place Criterion is a bridge between the above nonstochastic decision 
rules and the use of probabilities. It is based on the "principle of insufficient reason, " 
first formulated by Jacob Bernoulli, who stated that if there is no evidence that one 
event (state- of- nature) is more likely to occur than another, then the events should be 
judged equally likely. 

La Place stated that we should take the arithmetic average of the utilities which 
is equivalent to Bernoulli's principle. 


Thus our decision rule in this case is to choose that action a. which gives 



This, in effect says that each state-of-nature has a probability l/n. 

Applying the La Place Criterion to the stated problem in Figure 7-2 we have 

La Place 


8 

6 

8 

6 

8 
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7. 4 SUMMARY OF USAGE OF DELPHI- OBTAINED DATA 

The Delphi procedure is one of the most efficient known for uncovering the 
implicit models that lie behind opinions in the "soft" areas. An industrial or govern- 
mental Delphi effort is sponsored to collect opinions of individual experts and combine 
them into judgments that have operational utility to policy-makers. What to do with 
such judgments, particularly when dealing with the future, involves the planning cycle 
and decisionmaking. Planning and decisions assume a model of the situation, recog- 
nized or not, intuitive or well-defined. All our decisions imply some assumptions 
about the future; if assumptions are not made explicit, we do not criticize them. The 
Delphi method may be used to identify the elements involved, whether technological, 
societal, economic, values, assumptions, or decision criteria. The administrative 
plan for future-oriented Delphi application must anticipate the form of results and 
plan its utilization for any degree of effectiveness of the forecast. 

The references and bibliography are sources of much additional information 
about Delphi methods, technological forecasting, decisionmaking, and the telecom- 
munications future (see also Appendices B and C) which may be perused for amplifica- 
tion of information presented in this handbook. 
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APPENDIX A - MATHEMATICS OF DELPHI METHODS 
A . 1 INTRODUCTION 

One of the advantages of the Delphi method over conventional conferences is that 

not only is a consensus opinion produced, but also numerical data is produced on the 

spread of opinion among the participants after they have considered all the arguments 

that have been raised. In situations where Delphi panel estimates could be checked, 

it has been found that a small spread among the panelists was associated with median 

estimates that were only slightly in error from the true facts which the panelists were 

guessing at. When the panel members had a relatively large difference of opinion after 

several rounds of the exercise, their median estimates were, in general, less 
7 

accurate. The measure which is usually used in Delphi exercises to express the 
spread of opinion within the group is the interquartile range. It and the median are 
defined in Paragraph A. 2. 

A conventional conference places the participants under considerable pressure 
to arrive at a consensus. Sometimes there are two or more groups of participants 
who have quite different views on a question. In such situations, if the participants 
remain divided after hearing all the arguments, the matter is usually settled by a 
majority vote. Sometimes, if the minority feels strongly enough about the matter, 
they may write a minority report, but such action is infrequent. In a Delphi exercise, 
on the other hand, the existence of such blocks of opinion can be made known to the 
people using the results of the exercise. Such information may alert them to possible 
political or other opposition to the majority position, and will at least cause them to 
examine the situation with more care than they might use if the panel had been more 
nearly unanimous. To provide this information to the people using the results of the 
exercise, the actual distribution of the panel results, or an approximation thereto, 
must be part of the output information, in addition to the median and interquartile 
estimates. Such a distribution is easiest to comprehend if presented graphically. 
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Cross-impact analysis is a technique which can be used to check whether the 
probability estimates generated by the participants in the exercise are consistent with 
their estimates of conditional probabilities. If not, the inconsistencies can be brought 
to their attention, which will produce deeper, and presumably sharper, thinking. The 
basic cross-impact analysis procedure is given in Paragraph 5.9. Background infor- 
mation on the advantages of various methods of doing the cross-impact analysis is 
given in Paragraph A. 3. 

It is frequently of interest to know how sensitive the probability or date of 
occurrence of one event is to the occurrence or nonoccurrence of another event. 
Paragraph A. 4 briefly discusses the procedure for dealing with this problem. 

A. 2 MEDIAN, INTERQUARTILE RANGE 

The median of a distribution is that value of the variable such that half the mem- 
bers of the distribution are larger, and half are smaller. Of course, in those cases 
where there are ties, there may be no member such that half the observations are 
larger and half are smaller. Also, there may be a whole range of values such that 
half the observations are larger and half are smaller. The procedure below can be 
used to obtain an unambiguous value for the median in every case. 

When there is an odd number of observations, the median is the middle obser- 
vation in this list. (The observations must be listed in order of increasing magnitude. ) 
When the number of observations is even, the median is midway between the two cen- 
tral items. 

Just as there is one median which, in effect, divides the distribution into two 
halves, there are three quartiles or points, which divide the distribution into four 
quarters. These three quartiles are usually designated by the following set of symbols. 

Qj = First quartile 

Q 2 = Second quartile = median 

Qg = Third quartile 
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If there are N observations, and if N/4 is an integer, the first quartile has the 
value halfway between the N/4th observation and the next observation. The third 
quartile has the value halfway between the 3N/4th observation and the next observa- 
tion. For example, if N = 8, the first quartile is halfway between the second and 
third observation, and the third quartile is halfway between the sixth and seventh 
observation. 

If N/4 is not an integer, the first quartile has the value of the observation whose 
position corresponds to the next higher integer above N/4, and the third quartile has 
the value of the observation whose position corresponds to the next integer above 
3N/4. For example, if N = 9, Q = 9, Q = the value of the third observation, Q = 

XX O 

the value of the seventh observation. 

The interquartile range is the range of values of the center half of the population, 
namely, Q g - Q . 

A. 3 CROSS-IMPACT AND CONTINGENCY ANALYSIS 

It might be thought that a good procedure for obtaining estimates of the proba- 
bilities of various events is one which produces a consistent set of probabilities and 
requires a minimum input of information from the participants in the exercise. Such 
is not the case, however. If it were, cross-impact analysis would have no role, since 
the participants could simply stop after estimating the probabilities of the individual 
events they are asked about. In the absence of further information which the partici- 
pants could have supplied, this list would be consistent, and it would be much simpler 
to obtain than the data required for cross-impact analysis. 

Further thought reveals that people may be able to make better estimates of 
conditional probabilities than of overall probabilities, at least for some items. For 
example, one individual may not have a good estimate for the likelihood of event A 
occurring, but he may have a good feel for the likelihood of event B. Another may 
have a good estimate for the likelihood of event A if B occurs, and the likelihood if 
B does not occur, but not have a good feel for the likelihood of event B. By putting 
these two estimates together via equation (1), it may be possible to obtain a better 
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estimate of the probability of the occurrence of event A than either of them could 
have produced alone. 


where 



P(A) = P(B)P(A/B) + P(B)P(A/B) 


P(A) 

= the probability that event A occurs 


P<B) 

= the probability that event B occurs 


P(A/B) 

= the probability that event A occurs, 
B occurs 

given that event 

P(B) 

= the probability that event B does not 
P(B) = 1 - P(B). 

occur. 

P(A/B) 

= the probability that event A occurs, 
does not occur. 

given that B 


( 1 ) 


Even if the conditional probabilities cannot be estimated more accurately than 
the overall probabilities, use of equations like equation (1) can reveal inconsistencies 
that can promote further thought and clear up mistakes. If someone estimates indi- 
vidually the five numbers involved in equation (1), and finds the equation does not 
balance when he plugs them in, he can detect and correct errors in his thinking he 
would not otherwise be aware of. Cross-impact analysis is a procedure for doing 
this with a large number of interacting events. It checks whether the overall proba- 
bility estimates are consistent with estimates of conditional probabilities. If they 
are not, it attempts to modify the probabilities to make them consistent with the condi- 
tional probabilities. 

Not only does P(A) have to satisfy equation (1), but it also has to satisfy equation 
(2), where C is some other event whose occurrence affects the probability of A 
occurring: 

P(A) = P(C)P(A/C) + P(C)P(A/C) (2) 

If there are N events whose probabilities are to be estimated, each of the N 
probabilities must satisfy N-l equations like equations (1) and (2). Changing the 
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probability of any one event to balance one of these equations will affect the balance of 
N-2 other equations. Consequently, even if the conditional probabilities are correct 
it is not simple to obtain a consistent set of probabilities for the various events. 
Furthermore, as Dalkey shows in Reference 38, the conditional probabilities may 
themselves be inconsistent with each other. Dalkey points out that there is no "correct" 
method of resolving the inconsistencies. As he says, there are several directions 
that can be taken, depending on the interest of the study manager, on the availability 
of respondents for reestimation, and the like. "Assuming no restrictions on reesti- 
mation, it appears desirable to present the information concerning inconsistencies 
to the respondents, and obtain reestimates from them. " He gives a method for 
averaging out the inconsistencies in the conditional probabilities, and then uses these 
to generate a consistent set of event probabilities that will be as close as possible to 
the original probability estimates. The main disadvantage of his method, however, 
is that it requires conditional probability estimates for all pairs of events including 
pairs that we are not accustomed to thinking of as dependent. For example, if event 
A is the development of a 100 kW satellite power supply by 1980, and event B is the 
initiation of direct satellite-to-home TV broadcasts by 1990,, it is easy to think about 
the probability of event B given that event A occurred, but it is jarring to think about 
the probability of event A given that event B occurred, and the probability of event A 
given that event B did not occur. Dalkey’s method requires such estimates. We are 
not used to estimating the probability of a prior event if. a subsequent event occurs. 
Therefore, such estimates are likely to be much more erratic than the ones we are 
used to making, and this will almost certainly impair the results of a procedure using 
such estimates. 

A procedure that seems reasonable is to use a. reiterative Monte Carlo simula- 
tion, in which the initial estimates of probabilities of the various events are used to 
obtain an improved set of estimates, these are used as the starting point for a further 
improvement, and so forth, until the estimates stop changing substantially. At this 
point, if such a point is reached, the probability estimates would be consistent with 
the conditional probabilities. The Monte Carlo procedure does not require conditional 
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probabilities of earlier events given that later events occur, so the objection raised 
above to Dalkey's method does not apply to the Monte Carlo approach. Oddly enough, 
this procedure does not seem to have been used by any of the researchers who have 
written on the cross-impact method. Instead, most of them go through only the first 
iteration of this procedure, taking an original set of probability estimates and produc- 
ing an improved set, but not going on until the process converges to a stable set of es- 
timates. This occurs because they start from the same initial probabilities on every 
round of the Monte Carlo simulation, rather than starting from an updated set. 

A further difficulty which is encountered in the Monte Carlo procedure as it has 
been described in several of the papers, although not generally pointed out as such, is 
that if at some point in the simulation events B, C, and D have occurred, and E has 
not occurred, and the occurrence or nonoccurrence of event A is about to be deter- 
mined by picking a random number, there is a question as to what to do with the 
random number. We could elect to employ equation (1), and compare the random 
number with P(A/B), since we know that in this round P(B) = 1 and P(B) = 0. Or 
we could compare it with P(A/C), using equation (2). Similarly, we could compare 
it with P(A/D) or P(A/E). There is really no reason for picking one or another of 
these equations, since none of them are strictly correct. The probability of event A 
in general does not depend just on the occurrence of B or C or D or E, but on all of 
them. The cross-impact matrix does not contain enough information to show these 
simultaneous interactions. Several authors use a random procedure to pick one pre- 
decessor event and the associated conditional probability. Others let all of the pre- 
cedent events which have happened or not happened affect the probability of A, by 
multiplying the odds of A by appropriate factors that depend on the occurrence or 
nonoccurrence of all prior events which affect A. 

For example, suppose that events B, C, D, and E always occur (or fail to 
occur) prior to the occurrence of event A being decided. Before we know whether B, 
C, D, or E will occur we estimate that the odds for event A occurring are 1:1 (prob- 
ability 0.5). We also estimate that if event B occurs it will double the odds for event 
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A, but if event B does not occur it will reduce the odds for A by only 10%. We also 
make similar estimates for the effects of the occurrence or nonoccurrence of events 
C, D, and E. During the Monte Carlo round being used as an example herein, the 
odds for A would be multiplied by the factors corresponding to the occurrence of 
events B, C, and D, and reduced by the factor corresponding to the nonoccurrence 
of event E. If these factors happened to be, for example, 2, 3, 1.5, and 0.2, the 
odds for event A would be 1. 8:1, corresponding to a probability of 0.643. A random 
number between zero and one would be selected and compared with 0. 643. If the 
random number were smaller than 0.643, event A would be considered to have 
occurred, and the simulation would proceed on that basis. If the number were larger 
than 0. 643, the simulation would continue on the basis of event A not occurring in 
that round. 


Probability of occurring can be computed from odds for occuring as follows: 


Probability 


Odds 

1 + Odds 


Similarly, odds can be computed from probability as follows : 

oddg = 

1 - Probability 

The above method of allowing for the effects of several precedent events on a 
subsequent event is believed to be better than the approach of only considering one 
randomly selected precedent event, but it, too, has its problems. In particular, one 
must be cautious not to double count cross-impact effects. For example, suppose 
event A, when it occurs, greatly increases the probability of events B and C occurring. 
It, of course, does not follow that the occurrence of event B has a positive impact on 
C, but there may be a temptation to think so. If one did enter data showing cross 
impacts on C from both A and B, when in fact there was an impact only from A, the 
impact of A would have been counted twice. 

Several authors have attempted to use rather complicated quadratic equations to 
compute the effect of one event on the probability of another, allowing for the time 
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39 40 41 

between them and the strength of the cross-impact effect. ’ ’ By multiplying the 


probability of an event by some factor which depends on the occurrence or nonoccur- 
rence of another event, the strength of the coupling, and the time between them, one 
can avoid the criticism made above that only a single predecessor event affects the 
probability of a successor event in a single Monte Carlo round. The probability of 
the successor event may have been multiplied by many factors by the time the trial is 
made to determine whether it occurs in a particular round. 

39 

A description of the procedure is given below: 

Given that development i has ’occurred’, the revised probability of occurrence 
for development j is 


P.’ 

1 


where 


p i 


p; 

i 


i] 


P j’ 

the probability of occurrence of development j by time t, 
prior to the occurrence of development i, 

the revised probability for development j, 

the element on the cross impact matrix indicating type and 
strength of the impact of i on j, 

the original estimate of the time for occurrence for develop- 
ment i with 50% probability, and 


( 3 ) 


t = 


the specified year in the future for which the modified 
probabilities are sought. Both the t and t^ are measur 
on a time scale with the origin at the present date. 


The actual computation of the revised probabilities for each of the developments 
proceeds along the following lines: One of the N developments is selected at random, 
and by drawing a random number and comparing it with the chosen development's 
initial probability of occurrence it is determined whether or not the development 
'occurred'. If the chosen development did occur, the probabilities of all of the 
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remaining developments are revised according to the previous formula, and the 
occurrence is recorded. ' If the chosen development did not occur this is also noted, 
and the probabilities of the remaining developments are left unchanged. Now one of the 
N-l remaining developments is randomly selected and it is tested for occurrence, and 
if it does occur the probabilities of all remaining developments not yet tested for 
occurrence are again revised. Now one of the remaining N-2 not yet tested develop- 
ments is randomly tested for occurrence , and the appropriate probabilities are again 
revised. This process continues until all N developments have been tested for occur- 
rence. This process of one complete evaluation of the N developments will be referred 
to as being a trial. By running a large number of trials, and keeping track of the cum- 
ulative number of ’occurrences' for each development, the proportion of occurrences 
out of the total number of trials results in the modified probabilities for each develop- 
ment. 

In the preceding description of the computation process it was assumed that 
during a trial any of the previously unselected developments had an equal probability 
of being chosen to be tested for occurrence. In many applications some of the devel- 
opments may be necessary predecessors of others, and hence random selection is 
incorrect. In such a situation the procedures for selection of developments for test- 
ing can be modified to reflect precedence constraints. 

There are several objections to this approach, however. First of all, the 
probability of event j is changed if event i occurs, but is not changed if event i does 
not occur. This produces a biased effect, since the original probability estimates of 
events presumably allowed for the possibility of other events occurring or not occur- 
ring. It would therefore be more reasonable to change the probability of event j in 
one direction if event i has occurred, and to change it in the opposite direction if 
event i has not occurred. Secondly, to keep P ’ in the range of zero to one, it is 
necessaiy to limit s.. to the range of -1 to 1. If this is done, however, the maximum 
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impact one event can have on another whose probability is 0.5 is to change its prob- 
ability to 0.25 or 0. 75, depending on whether the first event occurs or not. If the 
probability of the second event is near zero or one, the occurrence or nonoccurrence 
of the first event has even less effect on it. The method therefore lacks the flexibility 
necessary to model situations in which the occurrence of one event can change the 
probability of a second event from a low number to a large number. Finally, the 
inclusion of a term proportional to time implies that the method will be used with 
time varying in some way, with a* computer evaluating the probabilities at various 
points in simulated time. This, in turn, implies that the probability of event j occur- 
ring at that time should be calculated. But, the computer cannot evaluate equation (3) 
at any time other than the time associated with event j in the heading of the cross- 
impact matrix, because P is defined as the probability of occurrence of development 
j by time t, and the only value of t for which value of P. is defined is the one in the 
heading defining event j. Therefore, the inclusion of a term proportional to t in the 
equation is a spurious refinement. In the Monte Carlo simulation development i can 
occur only in year t., and development j can occur only in year t.. Since these years 
are given in the headings of the matrix, the effect of time can simply be included in 
s„, making equation (3) considerably less impressive looking.. 

It should also be noted that if the intent of this procedure is to permit Monte 
Carlo simulation in which events i or j could occur at several different possible dates, 
even graver problems would be encountered. In such a case it would be necessary to 
calculate the probability of event j occurring at a specific time (such as in a specific 
year). Equation (3) gives the probability that event j will have occurred by a specific 
time, which is a very different thing. 

There is a way of simulating the probabilities of events having a possibility of 
occurring in a range of years. This procedure is explained in Reference 30. In that 
reference Helmer describes a game based on this approach, with a matrix showing 
the effects of individual events on individual events, individual events on each of a 
set of trends, trends on events, and trends on trends. In Helmer' s notation an event 
is something which can happen only once, such as a particular scientific breakthrough. 
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A trend is a gradual development having a magnitude at any moment, such as GNP, 
population, etc. Part of the input to Helmer's game is a set of probabilities, P„, 
for each event E^, giving the probability that event E. will occur during time slot j, 
provided it has not already occurred sooner. There are corresponding projections 
for the trends. These additions make it possible to do a more realistic simulation 
than the previous approach permits. Helmer's procedure seems to be the most real- 
istic approach available at present, and is recommended if time and resources per- 
mit its use. Although Helmer seems to have used manual and graphical methods in 
his gaming, it would be easy to computerize the procedure, making it faster and more 
convenient. 

As Helmer himself points out, his approach is still highly simplified as com- 
pared with what a realistic simulation would include. For example, the effect of one 
event (E ) on another (E ) is considered to be characterized simply by a time delay 

X z 

before the effect is felt, (always assumed to be at least one time unit) and the factor 
by which the odds for occurrence E 2 are multiplied from then on. If E^ fails to occur, 
the (delayed) effect on the probability of E is in the opposite direction, with the same 
magnitude. A more realistic simulation would permit changing effects with time, and 
differences in magnitude for occurrence and nonoccurrence. These refinements could 
be added to the simulation fairly easily. 

The effects of trends on each other in Helmer's approach are actually rather 
similar to the KSIM simulation, discussed elsewhere in this report. Helmer's games 
are more realistic, in that the trend variables are quantified in realistic units appro- 
priate to the variable, such as dollars, channels, telephones, etc. Furthermore, 
Helmer includes time delays. In KSIM all the variables are arbitrarily scaled from 
zero to one, or from zero to 100. On the other hand, the KSIM simulation is easier 
to set up. A synthesis of these two approaches could be useful. 

Prior to aggregating the probability estimates of the participants and performing 
a Monte Carlo simulation, it might be useful to give each participant a printout of his 
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probability and conditional probability estimates and their implications in terms of 
equations like (1) and (2), after which he would have one or more opportunities to make 
his estimates more consistent. While a more consistent set of probabilities and condi- 
tional probabilities is not necessarily a more accurate set, one is sure that something 
must be wrong in an inconsistent set, whereas a consistent set at least has the pos- 
sibility of being correct. Some participants, particularly those not familiar with 
probabilities, may get bogged down in such a procedure, so it should be made strictly 
voluntary. This procedure does not seem to have been employed in any of the cross- 
impact studies reported in the literature. 

A. 3.1 Simplification of Matrix 

As mentioned in Paragraph 5.8, the cross-impact matrix can be made less for- 
midable to think about and work with if it is separated into submatrices that do not 
affect each other, if there are such submatrices. The procedure for doing this sep- 
aration is quite simple. Suppose the events defining the row and column headings are 
called E^, E^ .... E^. Starting with E^ write down the subscripts of all the 
events that E^ affects or that affect E . These can be found by inspection of the 
original matrix. If there are no events that E^ affects or that affect E^, E^ is an 
isolated event and can be segregated. If there are some events, list them, and for 
each of them write down the subscripts of the events which they affect, or which 
affect them, if they have not already been listed. Continue this until no new events are 
added to the list. The events on the list form a connected group, and can be formed 
into their own matrix. If there are some events left, repeat the procedure, starting 
a new list with any one of them, to get the next group of interrelated events which 
form their own matrix. Repeat until no events are left. You may now have several 
small submatrices and several isolated events, rather than one large matrix to work 
with. In some cases, of course, it will be found that all of the events in the original 
matrix are connected. 
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A, 3, 2 How Many Monte Carlo Trials Should Be Used? 


In performing a Monte Carlo, simulation, one has to decide on the number of 
Monte Carlo trials to use. Increasing the number of trials reduces the statistical noise 
in the answer, but the noise is inversely proportional to the square root of the number 
of trials, so once an adequate level of accuracy is reached,, additional accuracy rapidly 
becomes more expensive in terms of computer time. 

If an event has a probability p of occurring in a Monte Carlo trial, and a proba- 
bility q of not occurring, where q = 1-p, and if N trials are made, the standard devia- 
tion in the fraction of time the event will occur during the N trials is \/ pq/N. The 
number of occurrences actually has a binomial distribution, but for most purposes 
this can be approximated by a normal distribution. In a normal distribution, one 
is 68 percent sure of the result being within one standard deviation of the expected 
value, 95 percent sure of being within two standard deviations, and 99. 7 per- 
cent sure of being within three standard deviations. For example, if p = 0. 5 and 
N = 1000, the standard deviation = 0. 0158. If the expected value is 0.5, therefore, 

1000 Monte Carlo trials will give a 95 percent probability that the computed result 
will lie between 0.468 and 0. 532. One would have a 99. 7 percent probability that the 
results will lie between 0.453 and 0. 547. This is sufficient accuracy for the purposes 
discussed herein, since the input data is highly unlikely to be more accurate than this. 

If the true probability were any number other than 0. 5, the standard deviation 
would be even smaller. For example, if p = 0.1 and N = 1000, the standard deviation 
would be 0.0095. The simulation results would then have a 95 percent probability of 
being between 0. 081 and 0.119, 

On modern computers, 1000 Monte Carlo trials should take no more than a 
few seconds of time, and so 1000 trials are recommended as a reasonable run from 
the standpoint of accuracy and cost. 

If p is extremely small, so that Np would be less than 5 with N = 1000, either 
the number of trials should be increased until Np = 5 or the Poisson approximation to 
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the binomial distribution, rather than the normal approximation, should be used. The 
Poisson distribution states that the probability of getting exactly r occurrences out of 
N trials is 


P(r) 


-Np 

e 


IKE)! 


r! 


As has been mentioned, if the original estimates of the probabilities of the 
various events were consistent with the estimates of the cross-impact effects, the 
results of the Monte Carlo simulation should agree closely (within the statistical 
noise) with the initial probability estimates. If one or more of the probabilities 
differs by more than two or three standard deviations from the original probabilities, 
there was probably an inconsistency. There is no reason to believe that the results 
of the simulation are necessarily consistent with the cross-impact effects, though 
they should be closer than the original estimates were. The Monte Carlo run should 
be repeated, using the new set of probabilities as a starting estimate, to see if further 
substantial changes occur. Presumably the process will eventually converge to a 
consistent set of probabilities. The necessity for performing such iteration seems to 
have been overlooked by all previous practitioners, so it is advanced with some 
diffidence. If the process does not converge, the rationale for using cross-impact 
analysis would be weakened or destroyed. Even if convergence does occur, there is 
no mathematical proof at present that the solution will be unique. In spite of these 
caveats, cross-impact analysis is believed to be a useful and worthwhile procedure. 


A . 4 SENSITIVITY ANALYSIS 

It is frequently of interest to know how sensitive the probability of occurrence of 
one event is to the occurrence or nonoccurrence of another event. Reference 31 
defines the sensitivity of dependent event A to conditioning event B as the absolute 
value of (P(A/B) - P(A/B)]. In the Delphi exercise reported on in Reference 31 these 
conditional probabilities were estimated by the panelists. The value of each sensitivity 
factor of interest was computed from the above expression, and the median and quartile 
values of sensitivity were computed and published as part of the results of the exercise. 
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However, sensitivity can be given a slightly different, and perhaps more useful 
definition. It is possible that the occurrence of event C could affect the probability of 
occurrence of event A. The probability of occurrence of event A would be sensitive 
to event C in such a case, but this might not be revealed by the definition given pre- 
viously, since the participants in the exercise might think in terms of first order 
effects when estimating conditional probabilities. It would therefore be preferable 
to find the sensitivities by running the Monte Carlo simulation of the cross-impact 
matrix twice for each event; once with the event given a probability of one, the other 
time with a probability of zero. The same random number seed should be used each 
time, guaranteeing that any differences in the results are caused by the sensitivity 
effects, rather than by differences in the statistical noise. For example, if event B 
is given a probability of one in the first run and a probability of zero in the second 
run, the sensitivity of each other event to event B can be found by subtracting the 
probabilities produced in the second run from those produced in the first run. Further- 
more, it is a good idea not to take the absolute value, since the sign of the sensitivity 
is also of interest. 

If one wishes nevertheless to use the approach given in the first paragraph 
rather than the procedure recommended in the second paragraph, it will be necessary 
to have the conditional probabilities. The procedure recommended in Section 5 of 
this report did not use conditional probabilities in the cross impact matrix. Instead, 
it used the factors by which the odds for the occurrence of one event would be multi- 
plied if another event occurred. These factors can be converted to conditional prob- 
abilities by using the overall probabilities and the procedure given in A. 2. For 
example, if the probability of A is 0.75, and the occurrence of B multiplies the odds 
for A by a factor of 2, one would convert the probability of 0. 75 to odds of 3:1, 
multiply by two getting 6:1, and convert this back to a probability, getting 0. 857. 

The value of P(A/B) would thus be 0. 857. 
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APPENDIX B - SUMMARY OF THE TELECOMMUNICATIONS FUTURE 


Both the selection of panel members and the delineation of significant questions 
for them to consider depend primarily on the present status of national communications, 
and on those individuals and entities responsible for continuing telecommunications 
development. 

First, consider some of the major areas that are embraced. These areas are 
not mutually exclusive. The questions to be initially generated for the panel of 
specialists should be distributed among the identified areas of telecommunication 
needs in such a way as to assure sufficient coverage. 

The forecast of national communications needs, like all forecasts, must evolve 
from a knowledge of potential needs and of present trends in technology. To examine 
in detail all the possible avenues of progress is beyond the scope of this appendix. 

What can be done, however, is to crystallize some of the thoughts on the subject and 
give the reader a structure, a frame of reference, which he can use to better formu- 
late the answers to Delphi questions. Hopefully, this appendix will help in phrasing 
pertinent questions, and help the panelist focus on ideas related to the Delphi inquiry 
in order to add dimension and weight to his answers. 

Communications is inherent in the nation, society, and business world activities — 
almost like the air we breathe. If it is of good quality and plentiful, we accept it as a 
given blessing. Our nation, is so dependent on fully functioning communication networks 
that we can hardly imagine being deprived of any existing system. A first reaction 
answer to, "what are this country's telecommunications needs ?" might very well be, 
"more!" When asking experts or informed persons what are the national communication 
needs about the year 1990, we are, in effect saying, "please look at a fabric of national 
life that is feasible and desirable in your view, and give your opinion of the communica- 
tions goals and values that our country should have at that time. " 

Focusing on the future and then moving from the future to the present to examine 
how one may obtain that future is the normative approach to forecasting. However, • 
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this is but a single path into the future. For a forecast to be meaningful, it is vital 
that an exploratory as well as normative approach be taken and that the two be linked 
in a cybernetic feedback fashion. This means that, while the past is recognized as 
important in establishing the future (exploratory), new futures are considered as a 
basis for changing the trend (normative) . In the remainder of this appendix, the 
normative approach to communications needs is first discussed followed by the 
exploratory approach. 

There is communication when there is a transfer of information from user A to 
user B and conversely although not necessarily so, from user B to user A. This 
transfer of information has for a long time been assumed to take place between human 
beings. This assumption is no longer correct and we must take into account nowadays 
three types of communication: man-man, man-machine, and machine-machine. 

The technology that provides existing means of transfer of information is the 
backbone upon which future technology solutions will evolve. * Future technological 
solutions will be thought of in terms of goals that are desirable, such as a lowering of 
cost (decrease of cost of earth satellite stations), an increase in the quantity of people 
that benefit directly or indirectly from the development (number of voice circuits per 
link) or other criteria directly linked to efficacy, efficiency, time, and the scarcity of 
resources. 

The exploratory examination of all possible avenues and trends brought forth by 
existing or projected technological means will be monitored by considerations of an 
ethical or social nature. This is made necessary by the all encompassing nature of 
technological developments in our times. These ethical and social considerations will 
thus include behavioral, moral, religious, ecological, and energy considerations, as 
perceived by each panel member. The questionnaire should reflect this preoccupa- 
tion with values by seeking information about the desirability of technological issues. 

*It is conceivable that some unknown technological development will be translated into 
means to transfer information. However, we are concerned here with known or 
intuitively possible means to achieve that transfer. 
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The foregoing reflects a first level, or conceptual level, of the problem at hand. 
At this point it is worthwhile to .examine how all topics to be covered in the forecast 
will evolve from issues relative to information, processing, storage and retrieval of 
information, transmission of information, sources and sinks of information, control 
of information and, to an increasing degree, privacy of information. 

The need for information has constantly been increasing because of the increas- 
ing complexity of modem life. This has also had the effect of reducing the required 
reaction time associated with day-to-day events. The payoff and risk of decision at 
government, corporate or private levels has become greater, thus justifying the • 
gathering of information for statistical purposes. 

Information in its raw form is but the beginning of the information process that 
leads to decisions. . The information that is being gathered manually and automatically 
must be suitably processed and stored. It is at this stage that the computer comes 
to the rescue with implications that loom large in our forecast exercise: technological 
advances which affect the performance of computers will also affect our capacity to 
handle information. 

So far we have evoked a static picture of information being needed, gathered, 
processed, stored and retrieved. Next we view possible developments in the trans- 
mission of information, which are highly important to a meaningful outlook of the 
future. Here, we are dealing with purely technical problems and solutions, like the 
availability of spectrum, the various uses of the medium (satellites, microwave, 
cable, laser), the availability of two-way communication (interaction) as opposed to 
one-way (broadcast), the transformation of information (analog vs digital), and the 
routing of information (range and capacity of the network, switching) . 

The developments in the satellite field are particularly challenging for the fore- 
caster of communications needs. Larger available power on the downlinks means 
that ground terminals can be made smaller and therefore less costly. The prolifera-r,, 
tion of ground terminals, the availability of frequencies through extension of the 


B-3 



spectrum or super-position on existing bands, increased sensitivity in band selection 
and beam directivity, will enable satellites to reach more remote locations and play 
a useful role in the education and entertainment of users located far from urban areas. 
The use of satellites in mobile communications needs also to be explored (another 
consequence of the emergence of smaller, tactical-like, earth terminals). 

Transmission of information between users located at remote locations can be 
exercised to perform control functions. If the users are permanently located we 
open up the whole field of automatic process control. If one of the users is in motion 
we deal with traffic control as applied to various forms of transportation (by air, by 
land, by sea). 

The availability of mobile users and of frequencies allocated to this application 
raise questions of a technical and social nature, e.g. , Will the day come when each 
person carries his own portable telephone? 

Where the impact of information and communication technology will be felt the 
most is at the level of the user, man and machine. 

The machine user is the computer and the whole field of computer technology 
with its own forecast can be addressed. The trend in the computer field has been the 
parallel development of very large, centralized computers, acting as the brain of a 
network, and of mini-computers at user locations in the network. Mini-computers 
will increasingly assume more and more functions previously assigned to large 
computers, and will perform them in less space and at a cheaper cost. 

Man as a user has available to him a whole array of broadband terminals. One 
of the major questions to be resolved, predominantly a social and economic one, is 
the extent to which terminals will be allowed to be converted from a one-way or broad- 
cast type of operation to a two-way or participative type of operation. It is at this 
level that we encounter the most challenging ideas for the future: homes with com- 
puter terminals, home videophone terminals, consumer information retrieval systems, 
electronic education and voting, etc. The advent of interreactive terminals raises the 
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question of how far can communication replace transportation, particularly as it 
affects work trips and shopping trips. 

The increasing use of electronic means in the gathering and transmission of 
information raises the question of privacy. There are two aspects of privacy that 
should bear consideration. The first one is that of the individual or group subject to 
censors for a variety of purposes. The second aspect of privacy is that of the infor- 
mation itself as it travels from user-to-user. Privacy is but one of the nontechnical 
issues raised by an information hungry society. It is quite possible that new laws 
might have to be drawn up in answer to new problems created by future uses of 
communications. 

We have given the reader an overview of the type of issues that he will face in a 
communications oriented forecast, be he an evaluator or a panel member. These 
thoughts are put in a schematic form in Figure B-l. 

Let us turn now to what may be considered the practical level of communications 
developments. The services to be offered to customers are expected to expand in scope 
and complexity as efforts to speed up the delivery of Information increase. The basic 
two-way random access telephone network will most likely expand to accommodate 
random access between man and machines and between machines. New telecommuni- 
cation services are needed to fill the time gap that exists now between the delivery 
of mail and the completion of a telephone call. The telegraph could be replaced by 
one-way voice and one-way record services; MAILGRAM is already one step in that 
direction. 

MAILGRAM is a service whereby a teletyped message is transmitted to a post 
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office near the addressee and is delivered the following morning by the mailman. 

The American Telephone and Telegraph Company allows a customer, through its 

TWX service, to send and receive messages over a teletypewriter circuit, through 

the switched network. The Western Union Telegraph Company provides a similar 
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service to its customers, called TELEX which is international in scope. 
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Figure B-l. Forecast of Communications Needs 












The advantages of TELEX, TWX, private line teletype, and private line 
facsimile could be made available to each household. It is not too far-fetched to 
visualize a family of services that will combine mail delivery and the use of the 
telephone network. Added motivation for this surge in efficiency of communications 
is the need to save energy and to try to replace transportation by communications 
whenever this is possible. 

Technically, increasing demands for communication circuits are pushing the 
boundaries of the frequency spectrum toward the frequency light. Moreover, the 
need for low cost broadband transmission facilities is giving strong motivation to 
the application of fiber optics technology to broadband circuits within a city. This 
development is complementary to the use of pulse code modulation (PCM) digital 
voice channels in U.S. metropolitan areas. 


The PCM technique is combined with the time division multiplexing (TDM) 

technique to provide short-haul (10 to 25 miles) transmission of 24 voice-frequency 
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signals (T1 carrier system). As a concept of data transmission, the T1 line can 

carry a bit stream of 1. 544 Mbps consisting of either message channels or wideband 

. * 44, 45 
data. 


From a telephone plant point of view, the increased use of PCM in cities signals 
a change in the classical structure of the loop configuration (between the central office 
and the customer). We are seeing and will continue to see an increasing use of 
common equipment, common trunk groups, multiplexer systems, and concentrators 
between the central office and the customer. These developments illustrate the 
expansion of data communications and the development of facilities specially geared 
to cope with digital systems. Whereas, during the early years of the data communi- 
cations business, service was mainly provided by modems on analog voice channels 
(sometimes specially conditioned), there is a continuing push to develop a truly 
digital data system, with switching centers capable of handling data. 
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The need for higher interoffice trunk capacity that is foreseen for the metro- 
politan areas in the 1980s is a consequence of increased urbanization and of new types 
of services that customers will expect from telecommunications companies. 

The services to be provided will satisfy person-to-person, person-to-machine, 
and machine -to -machine communications. They will include one-way audio-visual 
communication from an information center to a user (alphanumerical information 
retrieval, distribution of television channels, etc.), the ability to store and forward 
messages in a format, code, and language compatible with the receiving terminal, 
abbreviated addressing, conference calling, controlled access and privacy during 
conversation, etc. 

A restructuring of the existing telephone plant is needed to accommodate this 
increasing number of functions without impairing the nature of the network. 

Digital transmission is an attractive solution to this problem because it provides 
a relatively economic way to carry simultaneously voice, picture, and data signals 
and it matches time -division switching. 

A new digital long-haul coaxial cable transmission system will be introduced 
in 1975 in Canada. This system, known as the LD-4, combines PCM signals on a 
TDM basis to provide 4000 voice circuits. This system could very well set the pace 
in the extension of the digital hierarchy for North America. 

Any examination of the national communications needs will therefore address it- 
self to the growth of digital systems and to the successful integration of modems to 
the analog plant. 

In 1968, in what is referred to as the Carterfone decision, the FCC ruled that 
the telephone company could no longer prohibit without cause the use of customer- 
owned modulation/demodulation devices and the interconnection of private systems 
for interstate communications.^ 

This official encouragement of competition has stimulated the emergence of 
specialized common carriers and satellite carriers offering their services to data 
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communication users. Thus, specialized carriers like MCI Communications Corpo- 
ration offer nonswitched point-to-pGint circuits for intra-company communications 
for both data and voice. The Datran Corporation offers a switched all-digital net- 
work for data only, enabling any of its subscribers to communicate with any other 
Datran subscriber. Datran also has a contract from the International Brotherhood 
of Teamsters to integrate and support a nationwide, communications network. 
American Satellite Corporation is implementing a specialized data network for the 
Air Force with the earth terminals located on the user’s premises. 


Western Union, Western Tele-Communications, Inc. and Space Communications 
Corporation are also in the domestic satellite field. Space Communications Corpora- 
tion provides bulk channels for users with communications requirements large enough 
to operate their own ground stations. Alternatively, it provides turnkey service to 
allow a user to transmit data to a carrier’s regional ground station and have the 
system carry the data from there. 

Similarly, the data communication utility industry is utilizing the latest advances 
of computer technology to offer computer communication networks adaptable to user's 
needs. Time-sharing networks have helped users meet their requirements without 
having to design and implement data systems of their own. Companies offering 
comprehensive network services are pl ann ing to use existing carrier facilities to 
provide computer communication services based on the packet-switching concept 
developed for the Advanced Research Projects Agency Network (ARPANET). 


These developments in turn spurred action from the established common 

carriers. Consequently, the Bell System is now offering a digital data system for 

short-haul and long-haul transmission. As far as the latter is concerned, the 1.5 

Mbps Data Under Voice system efficiently utilizes the bandwidth available under 

the message channels on existing microwave routes. The 6.3 Mbps T2 digital line 
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uses paired cable facilities. 

The ripples caused by the interconnection decision are by no means settled, , 
hence, the configuration of telecommunication and computer networks is in a state of 
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flux. Many of the services to be offered in the future will depend on economic, legal 
and tariff matters. However, regardless of whether specialized common carriers, 
or utility networks dominate the communication scene, the progress of technology 
and demands of users will accelerate the use of data in such applications as banking, 
process control, transportation, health care, retailing, etc. 

Part of the communication needs will originate from the military. We are 
already seeing some "fallout" from military technology into the commercial world 
(message switching). Plans are underway to convert the hybrid Defense Communica- 
tion System (DCS) into an all-digital one. Therefore, for the initial purpose of setting 
up a base from which to study national communication needs, all three areas (commer- 
cial, military, government) will have to be represented on the panel although not 
necessarily in equal measure. 

The projection of communication needs into the future is not solely determined 
by technological possibilities. Considerations of a social nature also spur the 
development of new methods of communications or applications of existing ones. The 
increasing use of broadband circuits within a city has already been mentioned. Also 
within a city, the various educational, instructional and cultural needs of local 
communities will be translated in a variety of special-purpose television programs, 
each beamed to a selective audience. The same idea may be carried over in remote 
areas along the lines provided by ATS-6, the direct-broadcast satellite recently 
launched. Although it is not clear at this stage what mix of Cable-TV, UHF TV and 
satellite communications will satisfy these needs, this is an example where social 
needs will spur the development of the technological applications necessary to their 
fulfillment. 

We have seen that the forecast of national communications needs is very much 
influenced by the present and future capabilities of the communications and computer 
fields. We have also seen that the social aspects of this forecast cannot be neglected. 
This is because our society has become sensitive to the tremendous influence of 
technology on our lives and to the scarcity of our resources. A social presence 
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during this Delphi exercise is therefore highly desirable to point out possible 
advantages and disadvantages of certain technical developments. This social 
awareness may be obtained by having social scientists represented among the experts 
and the evaluators. Another way is to seek answers from the panel on the social 
desirability of a given technical event. 
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APPENDIX C - LITERATURE SURVEY OF FUTURE 
TELECOMMUNICATIONS TECHNOLOGY 


The selected experts, who may participate in a Delphi exercise dealing with 
long-term national telecommunications needs, are presumed! to be highly knowledge- 
able in specialized areas of communications and generally knowledgeable in many 
others. However, the present pervasiveness of communications in American society 
and its potential are so immense that a survey of recent literature may be helpful 
in providing additional perspective and "previews" of future telecommunications 
developments. 

This literature survey includes a review of the President’s Task Force on Com- 
munications Policy - Staff Papers 1 through 8 (13 volumes), dated June 1969. Staff 
Paper 1, entitled "A Survey of Telecommunications Technology, " provides an overview 
of national telecommunications addressing the following nine specific subject areas: 

1. Introduction 

2. Common-Carrier Network 

3. Transmission 

4. Switching 

5. Local Loops (Distribution) 

6. Trade-offs Among Transmission Switching and Local Loops 

7. Terminals 

8. Mobile Radio 

9. Television Distribution. 

Paragraphs C.l through C.9 summarize, under the titles listed above, the sub- 
stance of Staff Paper 1 as it relates to future telecommunications developments. In 
addition, briefs of related subject matter were selected from other publications to sup- 
plement the paragraphs devoted to each subject area. In each instance the source of 
the brief is identified and listed in the bibliography. 
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C.l INTRODUCTION 


A wide range of opportunities exist for expanded and new services in tele- 
communications in the U.S. Video services can be expanded by adding TV communi- 
cations capacity to about 20 channels per community; picture phones are technically 
available for point-to-point, small-screen, video communications; and so-called 
"switched broadband" systems offer pictures comparable to commercial TV in size 
and quality for point-to-point communications, shopping, or entertainment services 
of unlimited diversity. Automated data processing is emerging as an important 
customer of communications services. The number of computer-related terminals 
attached to the common-carrier network is growing rapidly, which may include such 
services as information retrieval, teaching, banking and shopping. Spectrum conser- 
vation made possible by low power transmission and multichannel receivers may 
foster extensive customer usage of portable and vehicle-installed mobile radios. 

Cost reductions are essential in exploiting new and expanded communications 
services. Every home could have a data terminal, a facsimile terminal, a video 
telephone, and access to a switched broadband network if such capabilities are made 
less costly so that they can be available on a wide scale. Not considering technology 
for the moment, one can say that quite stable relationships exist between present 
communications services and population and national-income factors, i.e. , such 
variables as GNP, personal consumption expenditures, government expenditures, and 
population. It is important to note that communications is not the only category of 
consumer expenditure in which new products or services will be introduced. In 
future trends it is not expected that new communications services will capture a 
greater share of consumer expenditures than at present; some new services may 
simply be new ways of doing old things. Although communications needs may be pro- 
jected in many respects by relation to an economic model, the future course in com- 
munications is highly dependent on the future of technology - relatively much more 
speculative; however, technology which leads to reduction of costs in providing old 
services can be fairly well foreseen. 


C-2 



While costs of sending large amounts of information long distances are sharply 
decreasing, the outlook is much less favorable for dramatic cost reductions elsewhere. 
Costs of long-haul transmission systems may be substantially reduced by increased 
capacity provided by satellites, terrestrial microwave, and coaxial cables. The 
relative costs of switching, terminals, and local loops appear highly intractable to 
cost reduction. It is considered that the computer services business employing 
common-carrier networks will find that reductions in communications costs are 
unlikely to equal the rate at which the cost of computer processing and storage capa- 
city is falling (i.e. , by 50 percent every 2 years). Because cables used to distribute 
TV signals locally share some characteristics of long-haul coaxial cables, their costs 
are likely to fall. 

Since the present communications system is not ideally suited to teleprocessing, 
a dedicated network for data processing may offer cheaper communications and 
greater flexibility. A major drawback to establishing an independent digital system 
has been the loss of economics of scale from sharing transmission facilities with 
voice traffic. In the near future it may be practical to establish a separate digital 
network for data customers within a shared system. On the same local distribution 
loops used by telephone customers, data signals could travel to local digital switches. 
From there, they could enter directly dedicated channels on the system’s transmission 
lines, or a separate long-haul digital network, perhaps provided via satellite, by- 
passing the expensive multiplexing and routing facilities of the basic telephone network. 
It is probable that, while data terminals remain costly, the touchtone telephone instru- 
ment will offer occasional users direct access to computers. 

The whole evolution of telecommunications in this nation has great implications 
for policy. The widely varying requirements of teleprocessors require a wide choice 
of services and freedom to establish subnetworks as necessary. Common carriers 
can provide these needs to a limited degree by offering a larger selection of band- 
widths, but customization for data communications is likely to exceed the diversity 
that franchised common carriers can provide. Progressive policy by regulatory 
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entities would be necessary for implementation. The policy regarding the depreciation 
method for investment in new technology affects the price that will be charged for new 
services, which, in turn, bears upon the expected level of estimated demand. Accel- 
erated, economic-life depreciation, which projects a larger share of equipment costs 
in the system's early years, is the preferable method in a period of technological 
change. Another implication for policy relates to the fact that cost reductions are 
most needed in local loops and terminals. These two elements represent a growing 
share of network costs; significant overall cost reductions will depend on progress in 
these areas. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

Hammer, Carl, "Telecommunications," Data Management , Vol 12, No. 4, April 
1974, pp. 12-17. 

The developing relationship between computer and telecommunication technology 
is possibly the most important event of our times. Computers will control the 
immense communication switching centers and assist in managing the enormous capa- 
city of the new linkages into usable channels. Telecommunications in return, will 
make available the power of computers and the information in data banks to millions 
of users in remote locations. 

It appears certain that future circuits will be designed to transmit very high- 
speed pulse trains into which voice, television, facsimile, and other data will be 
coded and sent in a uniform manner. 

It is anticipated that the greatest impact of teleprocessing applications will 
be within the Federal Government. Likely the number of ADP systems will continue 
to increase rapidly during the coming years and most of the newly acquired systems 
will be communications-oriented. Thus the combined cost of computer and long-haul 
telecommunication services for the Federal Government of the United States is 
expected to reach levels approaching $10 billion by 1980. 
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This accelerating pace of teleprocessing developments will severely strain 
many segments of our national economy. The common carrier communication industry 
may be hard pressed to find the funds needed for the required plant expansion; other 
entities may insist upon the right to share in the growing market. 

We may even see the beginning of fragmentation of what heretofore has been an 
integrated, nationwide teleprocessing communication network; significant security 
implications may not be adquately addressed; and even the question of negative socio- 
cultural impacts may not be adequately taken into account. 

Such considerations point initially to the need for a higher degree of control, 
coordination and policy determination ait the Federal level. In part, this need arises 
from the very nature of teleprocessing itself in that it is a new and qualitatively 
different medium for information processing. 

We have already noted that teleprocessing is not just computers plus communica- 
tions but instead has a unique dimension of its own much in the way that hydrogen and 
oxygen combined produce a wholly new property — water. It is the synergistic fact 
that teleprocessing is greater than, and, different from the sum of its parts that 
presents us with a need to manage this emerging resource, also on the international 
scene. 

On the one hand, teleprocessing systems provide a powerful new mechanism 
for controlling, directing and managing complex organizations through the applica- 
tion of cybernetic concepts. Decisionmaking institutions, whether business or Govern- 
ment, can be viewed essentially as dynamic man-machine systems; and teleprocessing 
systems facilitate the management of such organizations within the context of a sys- 
tems concept. The organization is thus viewed as an integrated whole wherein each 
system, subsystem, and supporting subsystems are associated with the total operation. 

At the same time, however, the rapidly evolving teleprocessing technology chal- 
lenges the capability of management to effectively integrate and manage the new 
resource itself to meet the needs of the organization. 
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Bagdikian, Ben, The Information Machines: Their Impact on Men and Media. 

New York: Harper & Row. 1971. 

A distinguished journalist, Bagdikian spent 2 years at the RAND Corporation 
where he directed a study of the impact of future technology on the gathering and 
dissemination of news. Part of his RAND research included a Delphi poll on the 
future of communications. 

In forecasting the future technology of newspapers, Bagdikian is well aware that 
the present experts in newspaper technology— the manufacturers of typesetting 
machines and printing presses — may know very little about the coming technology, 
which will arise from computers and electronics. At the same time, today’s com- 
mercial broadcasters, committed to preserving present techniques, also may not be 
the best sources of information on the news media of the future. 

But from his research and questioning of people in many fields related to com- 
munications, Bagdikian believes that "the main lines of the future seem to be clear, 
or as clear as a look can ever be a whole generation ahead. " 

"Somehow computers will be involved in the storage, delivery and switching 
of popular communications. Somehow there will be additional capacity for the con- 
sumer in his home to receive a greater variety of information than he does now. He 
may be able to control the timing, content, and form of this information flow in ways 
not now available to him. " 

The Delphi panelists were virtually unanimous that every major step in the 
news process will be substantially changed by 1980, and after 1980 more radical 
innovations in whole systems will begin to take effect. 

"A major innovation in home communications will be a reactive system, with the 
individual consumer having the power to order specific content and receive it imme- 
diately. There is already an elementary reactive system: it is possible to order 
items by telephone and get a reply by telephone. One can ask for other telephone 
numbers, or the right time, or airline schedules, or taped weather forecasts, and 
the answer comes immediately through the return electronic link of the same telephone 
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line. A more advanced system would permit the consumer to signal out for what he 
wants and get an immediate televised response. 

"The panel believed that such a reactive system would be in normal use by 
1990 (the specialists thought 1987), when the consumer would be able to get what he 
asked for either on his TV screen, by voice, or in a document produced in his home 
electronically. 

"The consumer will order this information, at first, by Touch-Tone telephone, 
the pushbutton signaling system the telephone company is introducing throughout the 
country. " 

As home communications devices evolve, Bagdikian forecasts, they will serve 
not only entertainment, home education, and live public events, but information from 
all kinds of organizations. 

Bagdikian, Ben, "How Much More Communications Can We Stand, " The Futurist , 
Vol. V, No. 5, October 1971, pp. 180-183. 

An immense volume of communication is technically feasible by means of a 
new world of electronic gadgetry. But the flow of information provided by this 
mechanical ingenuity must ultimately lead to a human being. For all the millions 
of words and images that our cables and computers can deliver, the final recipient 
is a human being who can process about 250 words a minute, at most about 1000. 

There may be a physical and psychological limit to living with an exclusively 
abstract intake. Although communications are becoming increasingly efficient from 
a technological standpoint, their content is becoming ever more meaningless. Ad- 
vertising and propaganda are debasing our symbols so that there is a breakdown in 
real communications between different generations and different political groups. 

The solution may lie in the increase of noncommercial media that are not committed 
to playing the game of collecting mass audiences for sale. 

The basic American ethics of tolerance of dissent no matter how hateful is in 
danger of dying at precisely the moment when the authorities are about to inherit 
more powerful systems of mass communications. There is no point in having new 
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communications systems unless they are open and unauthoritarian. As usual, we don’t 
have much doubt that we can produce and operate the technology. And as usual, we are 
not so sure that we can put the technology to work for humane and creative purposes. 

The disparity between the capacity of machines and the capacity of the human 
nervous system is not a small matter in the future of communications. It has 
individual and social consequences that are already causing us problems, and will 
cause even more in the future. The human being of the near future probably will 
need as much sleep as he does today. He will spend more time absorbing abstract 
information than he does today, continuing the trend of past generations. But there is 
a limit that is important in a number of ways. 

Haddad, Jerrier A., "Computer Systems Technology," published in Technology 
Forecast for 1980 by Weber, E. , Teal, G. K. , and Schillinger, G.A., Van Nostrand 
Reinhold Co. , 1971, pp. 189-198. 

Mr. Haddad is an IBM vice president, and director of its SDD Poughkeepsie 
Laboratory. 

If now we are to look at what we expect our progress to be for the next decade, 
of necessity we must project the possibilities in a number of critical areas. Certainly 
we must look to improvements in hardware, in programming, in digital data com- 
munication, and in application ingenuity. These aspects are relatively easy to dis- 
cuss. 

Additionally, however, we must consider that as computers pervade the every- 
day life of more and more organizations and individuals they will affect societal 
relationships as well as legal and economic relationships. Clearly we cannot fore- 
cast to 1980 without these latter considerations — and clearly these societal, legal, 
and economic considerations are more complex and deeper in effect than those items 
which can be resolved in a laboratory. This is not brought up merely as a disclaimer 
to a technical forecast. Rather it is mentioned as a strong motivational element which 
can and will sway the intensity and direction of technical progress in the next decade. 
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A summary of assumptions regarding computers in the 1980 period are: 

• We will continue to see a roughly three to five times improvement in the 
best performance every five years. 

• We will continue to see a roughly three to five times improvement in the 
cost to the user of a given performance every five years. 

• We will see continued increase in the capacity of on-line storage and in its 
performance. 

• We will continue to see programming systems increase in functions and 
performance most probably by breaking the problem down into subsystems. 

• We will continue to see batch fabricated electronics allow less and less 
expensive data entry and terminal products. 

• We will see digital data communication improve in cost and reliability 
almost independently of distance. 

The whole purpose of applying a computer system is to make people more 
effective. As the new generation, trained in the capabilities of computers and 
accustomed to their usage, finds its way to leadership in the professions, govern- 
ment, and business, the role of computer systems in our lives will become more 
pervasive and more meaningful. 

The objective of a data processing system is to have the needed information at 
the right place at the right time in the right form. And that type of definition implicitly 
assumes communication as part of that system. 

Now, if you step from that spot over and say: But there is a requirement for 
nondigital data — communication— nonaccounting, nonnumerical, if you will — and now 
what's the role of a digital computer in that? Then I think you'll find that what these 
people say is absolutely true; that the digital computer of one form or another will 
play a very strong role in the management of and in the operational aspects of com- 
munications systems of the future. Whether or not, by the way, pulse code modula- 
tion or the equivalent is used. 
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C. 2 THE COMMON-CARRIER NETWORK 


For convenience analysts look at the common-carrier network as consisting of 
four functional elements: 

1. Transmission facilities 

2. Switching facilities 

3. Local loops 

4. Terminals. 

Most domestic communications traffic travels over the common-carrier tele- 
communications network; different users share a common set of facilities. Some 
services do not use all four elements. Mobile radio and TV distribution lie wholly 
or partly outside the network. Radar, navigation, and meteorology rely heavily on 
the same kinds of technology as the more traditional communications services and 
are likely to benefit from and contribute to technological cross-fertilization. 

System planners for common-carrier networks are charged to minimize cost 
of each of the four functions. Planners may substitute one function for another or 
change costs by changing the way a function is performed. Presently, it is recognized 
that some tradeoffs may be made between switching and transmission; between ter- 
minals and local distribution lines. In seeking to decrease cost of transferring 
information, planners deal with three principal technical variables: channel band- 
width, S/N ratio and coding efficiency. New technologies, which may achieve a 
least-cost, reliable way of transferring information, rarely affect only one variable. 
Usually there is improvement in one variable and degradation in others. The rate 
at which innovation may be introduced is heavily limited by the need to adapt new 
systems to installed equipment. Long useful lives and a high ratio of fixed-to- 
operating costs are characteristic of transmission, switching, and local distribution 
facilities. It is easy to see that past investment decisions inhibit future evolution 
in the common-carrier network. Any new equipment must be compatible or installed 
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by taking adaptive or insulating measures. The upshot of technological changes 
are that, although they may technically be feasible, the cost consequences are often 
prohibitive. 

Each of the four basic elements of the common-carrier networks is discussed 
in Paragraphs C.3 through C. 7. 

C. 3 TRANSMISSION 

After World War n the common carriers began to substitute coaxial cables for 
paired wires, which offered much more usable bandwidth with less signal degradation 
over long distances. At the same time, line-of-sight and wideband radio transmission 
links were introduced, using the microwave portion of the electromagnetic spectrum. 
Twisted pairs of wire have been virtually replaced for long-haul transmissions, but 
are still the primary medium between local exchanges and telephone terminals. In the 
near future, satellites are expected to be strong competitors for long-haul trans- 
mission, particularly for distribution services and variable-capacity routes. In the 
more distant future, newer facilities such as millimeter-wave guides and laser sys- 
tems may come into use where routes with densities of 100, 000 circuits are required. 

Coaxial cables are essentially free from spectrum limitations and promise large 
economics of scale and low unit costs. They have certain advantages over radio for 
fixed point-to-point services such as operation over a wide frequency bandwidth and 
shielding from external radiation; however, signals over cables are subject to power 
loss and cannot pass extremely high frequencies. On the other hand, millimeter- 
wave guides are planned in the future to provide transmission in the range above 10 
GHz for routes of 200, 000 voice circuits and CATV systems. 

Ocean cables show the same trends toward economics of scale. 

Microwaves (1-10 GHz), like wire transmission methods, are characterized by 
high fixed costs and large economics of scale. Capacity increases for this line-of- 
sight medium have been significant, yet costs well exceed those projected for the 
highest capacity cable systems. Start-up and maintenance costs have been difficult to 
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reduce. Building of roads to relay sites and regular deliveries of fuel are costly 
factors not likely to benefit from other technological developments in the communica- 
tions field. Largely because of multiplex terminal costs and spectrum crowding, 
microwaves have not been used for short-haul transmission of telephone or television 
signals. 

Communication satellites provide another line-of-sight transmission medium; 
the cost of satellite circuits are expected to drop sharply in the near future. The com- 
munication satellite is a single-node configuration characterized by: 

1. Circuit costs within a single satellite system are essentially independent of 
terrestrial distance within the coverage area. 

2. Dense routes have some advantage because earth terminal costs can be 
spread over a larger volume of traffic. 

3. Capacity among the routes a satellite system serves can be reallocated, 
whereas, terrestrial routes are fixed. 

4. The entire capacity of a satellite repeater constitutes a single trunk group. 

5. Satellites are particularly suitable for distributing a single, wideband (e.g. , 
television) signal simultaneously to many points. 

Synchronous satellites having improved equivalent isotropically radiated power 
(EIRP) and more directive antennas appear to be the most-cost-effective road for this 
transmission media. 

In the competition among long distance transmission modes, high-frequency (3- 
30 MHz) radio has suffered from quality and reliability problems, even on the long, 
thin routes where it is most competitive from the standpoint of cost. At present no 
one seems to foresee a great future for HF systems. 

As an alternative to present analog signal forms, digital transmission techniques 
offer superior signal quality and interference protection, particularly on long routes 
and for radio systems. Digital signals are relatively easy to amplify as compared to 
analog. Repeaters in a digital system ignore the noise and generate a clean signal 


C-12 



at each relay point. In an analog system there is no way to separate the original 
signal from noise picked up along the way. The inherent disadvantage of a digital sys- 
tem is the large amount of bandwidth required to reproduce analog signals. The band- 
width requirements for some analog signals (e.g., video, picture phone) may be 
reduced by redundancy- removal equipment. Digital systems are presently able to 
overcome the disadvantage of large bandwidth requirements only where: 

1. The transmission medium is not bandwidth limited due to spectrum availa- 
bility (e. g. , cable, wire). 

2. New transmission facilities are prohibitively expensive. 

3. Conventional analog methods of expanding circuit capacity or the built-in 
signaling capability of digital channels can be exploited. 

The dominant characteristic of transmission technology is the advantage of large 
trunk groups, which provide strong economics of scale. As the length and circuit 
density of trunk groups change, so do the relative costs of different transmission 
media. Sometimes the trade-off is between length and channel density. For U.S. 
domestic services, paired-wire cable dominates for fixed capacity routes of fewer 
than 500 circuits; microwave between 500 and 15, 000; coaxial cables between 15, 000 
and 80, 000; and wave guides above 80, 000 circuits. Expected growth in volume of 
long-distance communications will allow the telephone companies to make greater use 
of economical, high-density transmission methods. It is anticipated that voice traffic 
will triple by 1980 and that information sent as "data" may exceed that transmitted by 
voice. By that time Bell anticipates that 90 percent of its long-haul circuitry will be 
in coaxial circuitry. Two technological breakthroughs in the 1970s promise expanded 
use of digital systems; in the meantime, digital and analog system developments will 
be in parallel. The two breakthroughs anticipated are redundancy removal techniques 
and full realization of large-scale integrated circuitry. 

On intercontinental transoceanic transmission routes, communications satellites 
are likely to become the dominant mode by 1980. The alternative modes are HF radio 
and ocean cables. For normal international telephony and data services, HF radio is 
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unlikely to suffice except for the smallest, most isolated nations. Cables, however, 
provide technical benefits which are not easily quantifiable, such as reliability, no use 
of radio spectrum, and better quality for some services. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

"Communications: Optical, m-m Waves Offer Promise, " Electronics , Vol. 47, No. 

9, May 2, 1977 pp. 36-37. 

The wideband systems that will be transmitting large volumes of data at high 
speeds in the 1980s will combine optical and millimeter -wave links. That's the pre- 
diction of a study recently completed for the Air Force's Rome Air Development Cen- 
ter by Martin Marietta Aerospace, Orlando, Florida. 

The Air Force commissioned the study as an aid to planning future intrabase 
communications between rapidly deployed command posts and a central headquarters 
unit. But the authors say much of the report applies equally well to commercial intra- 
city links. 

The major conclusion is that optical links are superior to m-m wave links only 
for distances of 1 kilometer or less, according to Terry Duffield, Martin Marietta 
staff engineer. In a paper being presented this week at IEEE Southeastcon in Orlando, 
Duffield says that the cost and performance studies performed by RADC show that the 
two types of systems are about equal between 1 and 3 km, but that millimeter waves 
take over at longer distances. In a related paper, Martin Marietta's H. B. Muench 
adds that today's optical systems are ready for serious consideration as short links. 

Duffield's study, which projected cost and performance figures through the late 
1970s and into the 1980s, centered on systems with information bandwidths greater 
than 500 megahertz and all-weather reliability of 99.9%. The millimeter wave sys- 
tems were solid-state systems using bulk-effect devices for sources. The optical 
systems used light -emitting diodes for the short links, gallium -arsenide lasers for 
links in the 2- to 3-km range, and carbon -dioxide and neodymium -doped ytrrium- 
aluminum-garnet lasers for the longer links. 
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The recent price reductions for gallium-arsenide lasers is the basis for 
Muench* s conclusion that they could be used in communications today. Lasers that 
were selling for more than $100 six months ago, he says, now are available for less 
than $20. In his paper, he quotes a recent Army (Fort Monmouth, N. J. ) estimate of a 
$1, 000 cost per terminal for a gallium -arsenide system operating at 10 megabits per 
second. Muench says that this cost may be achieved by early next year. 

With such a system operating at 1. 5 megabits per second, Muench points out that 
it could be useful for setting up short links in a telephone system to avoid the need to 
lay cables for a Tl-type link, the Bell System designation for equipment that multi- 
plexes 24 voice conversations on one wire pair. Muench says that such a system 
could be useful in, say, new housing developments to bring voice signals to a central 
switching office, or for setting up emergency communications. 

Feldman, Nathaniel E. and Kelly, Charles M. "The Communications Satellite - A 
Perspective for the 1970s,” Santa Monica, California, RAND Corporation, Report 
No. P-4658, September 1971. 

Satellites will not play a major role in world electronic communications in the 
1970s without major new R&D investment and new international arrangements to sup- 
port developments. What must be done to make communications satellites compete 
more effectively with other forms of electronic communications? 

We would characterize the status of communications satellites as follows: 

• International long-haul and defense point-to-point systems are growing 
rapidly utilizing low -output -power satellites. 

• Domestic systems are stalemated by regulatory authorities in the United 
States and by high development costs elsewhere except in the Soviet Union. 

• Over-all research and development (R&D) effort is minimal for advanced 
pioint-to-point systems. 

• Only minor R&D effort is being applied to the development of high-powered 

i satellites and small ground terminals. Canada is sponsoring a 200-watt - 


C-15 



per-channel 12-GHz communications technology satellite, and there is 
some development of small mobile ground stations for operation with 
defense satellites. 

• Frequency assignments for high-power satellites are being considered by 
the International Radio Consultative Committee (CCIR). 

It is commonplace for the less developed countries to feel that their communica- 
tions problems are unique and that these problems are neglected by the highly de- 
veloped countries. It suggests that a type of satellite needed by advanced countries 
that spread over large land masses can serve the less developed countries as well. 

Under Congressional direction to de-emphasize point-to-point communications 
satellite activity when Comsat was established, NASA essentially abandoned the 
development of high -power satellites and then proceeded to de-emphasize all satellite 
communications as well. By limiting the funds for all such activities, NASA opted for 
slow and modest progress. At today's funding levels, decades will be required to 
achieve the technical advance called for by a 1967 National Academy of Sciences study. 
R&D supported by the United States created most of the technological base for Intelsat. 
With exceedingly modest further support since the early 1960s, progress has been 
commensurate. Western Europe's communications -satellite programs, moreover, 
have demonstrated negligible operational results. 

For satellites to play a major role in future world communications, the United 
States must press R&D leading to the more versatile high-power systems. In particu- 
lar, the U. S. Congress must be persuaded to support a major international cooperative 
program to make progress possible. No national or international commitment to such 
an R&D effort exists. A new international consortium appears necessary to support 
the development of high-power satellites, since such a program appears beyond the 
means or charter of any communications organization. 

Organizing an international cooperative program will be difficult and will take time. 
If NASA were to go forward with the necessary technical developments while the potential 
users organize, the major goal — to implement a high-powered reliable communication 
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satellite — would be advanced. Since bilateral national agreements on satellite com- 
munications have consistently proved easier to negotiate and more effective than multi- 
national agreements, strong U.S. leadership in negotiating a series of bilateral agree- 
ments may lay the groundwork for an international program. 

The primary motivations for the development of communications satellites in 
most countries have been to enhance national prestige, to support the aerospace in- 
dustry, and to secure the manifold benefits of technology for the nation at large. Over 
the past decade, this order has become inverted and social benefit has been added to 
the list. So we see a grave danger that frequency allocations, power-flux densities at 
the ground, and orbital-slot assignments may be allocated by international agreements 
that inhibit high-power satellite development. Broad frequency bands and some very 
high flux' densities need to be made available while delaying restrictions for many 
years to permit extensive experimentation with operational systems. Means must be 
found to deal with the pre-emption problem. Canadian and U. S. domestic 
communications -satellite operations and the U.S. -Canadian agreements which evolve 
might provide a firm experimental basis for international regulations. 

In summary, we expect international and domestic point-to-point communica- 
tions satellite systems to multiply throughout the decade. In developed regions, they 
will remain auxiliary to the over-all ground communication complex, even though very 
significant in dollar volume. In many undeveloped regions, however, point-to-point 
systems could become the primary means of communication. If potential users of 
high-power satellites make the necessary arrangements for frequency assignments, 
user agreements, financing, regulation, and legislation (both national and international) 
and establish appropriate interfaces with other communication entities and systems, 
then many new applications with large growth potential can be developed for populated 
as well as undeveloped regions. 

Without support on the order of $200 million per year to the development of the 
space segment, however, no dramatic new services or improvements in communica- 
tions via satellite will occur in this decade. Yet, among space activities, only 
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communication- satellite R&D offers the potential of generating a multibillion-dollar 
industry that can both make a profit and support world understanding. 

Hult, J. L. Broadcast Opportunities with Satellites and CATV, and their Control in 
the Public Interest . Santa Monica, RAND Corporation, Report No. P-3333, March 
1970. 

Satellites do not provide a communications capability that could not be achieved 
by other means. However, they can drastically change the economics of communica- 
tions both in magnitude and form and in so doing can serve as a catalyst for an enor- 
mous increase in volume and change in style of communications in general. It is the 
revolution in use of communications that can have the most far-reaching impact on 
every institution and aspect of society. It is for this revolution that we should prepare 
society, if we are to avoid serious convulsive reactions to the profound changes that 
seem inevitable. 

The term "communication” as used here is meant to include any transfer or 
exchange of information. It may involve primarily natural transfer processes, for 
example, listening to or looking at the information directly in a personal visit; it may 
involve electronic transfer of information, for example, by telephone, telegraph, 
picture phone, radio, TV, facsimile, or data links; and it may involve recorded infor- 
mation such as mail, newspapers, magazines, books, records, tapes, or photographs. 
Communication in this most general use of the term is the fabric of society. It links 
places, times, institutions, and individuals to form societies. It is the means on 
which we must depend if we are to learn about and adapt to the on-rushing change in all 
aspects of technology. And, yet the technology of communications itself is one of the 
most active and volatile of all the components of our society. Thus, we may find it 
very difficult to ensure the appropriate communications development that will foster 
an evolutionary, rather than a revolutionary, transformation of society. 

The pace of change is so dramatic that we need to focus attention both on the 
design of our material environment and on our philosophy of living, so that all elements 
of society can adapt gracefully to the changes. For example, we should design our 
communication equipment and its related facilities and economics for replacement or 
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obsolescence m 10 years or less, rather than freezing systems to obsolete configura- 
tions for most of their lives. Similarly, we need to review and reexamine basic values, 
concepts and traditions, such as various ’’freedoms" and "rights,” privacy, individualism, 
the family, education, monopolies, and the regulation of international resources to deter- 
mine if what has become conventional will be suitable to the changing environment. 

Hult, J. L. , Satellites and Future Communications. Including Broadcast . Santa 
Monica, California, RAND Corporation, Report No. P-3477, April 1967. 

The important factors of public interest and support, government regulations and 
policy, and special interest are likely to determine the rate at which the potential of 
communication satellites is exploited. Technology will neither set the pace nor 
seriously constrain the possible beneficial applications. Among regulatory and policy 
matters related to the rate of satellite communications development one may make 
arguments for eliminating monopoly extensions to terminal ownership, the importance 
of domestic satellite growth, and the economic control of spectrum usage. 

The large, versatile, high-gain antennas that can be boosted to, and erected in, 
the compellingly attractive environment of synchronous orbits are technologically 
feasible. The accurate, long-life control of these antennas in both attitude and orbit 
will enable us to increase satellite relaying capacity by tens of millions of times and 
multiply the effective available spectrum by hundreds to thousands of times. This 
could support an unprecedented increase in the application of electronic communica- 
tions. It could provide a relaying cost-rate competitive with the shortest microwave 
relay, a much more beneficial usage of the spectrum, and a variety of capabilities not 
otherwise available. It becomes economically attractive only when large capacities 
are used, in which case it may have a serious impact on current systems and invest- 
ments. The technology should also make possible satellite TV broadcast directly to 
homes at UHF frequencies with better quality signals at less broadcast cost than with 
conventional transmitters. However, transmission of broadcast programs with a gen- 
eral multipurpose broadband access to the home through central . exchanges may prove 
to be the most appropriate means of distributing the bulk of future broadcast-like pro- 
grams. 
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C. 4 SWITCHING 


Fundamentally, switching is a way of conserving communication lines. A net- 
work configuration has evolved where a group of terminals share a local switching 
center, which, in turn, is connected via a multi-channel trunk route to a wider-area 
switching center, and so on through, at present, a hierarchy of five levels of switches. 

The two basic switches are circuit and store-and-forward. There are three 
types of circuit switches: space-division, time-division, and frequency-division; 
each type provides an apparent continuous connection between communicating points. 
Store-and-forward switching at the simplest level is exemplified by a telephone 
answering service. Another example is the Western Union torn-tape system, which 
concentrates messages at a central location where they are stored on punched tape 
and relayed in batches. But, today, the high-speed computer is usually thought of 
for performing store-and-forward switching functions. Most rapid growth of such 
computer switching is anticipated as an ancillary service to data processing. 

Progress in circuit switching has brought direct distance and automatic dialing 
and has increased switching speed. The key to switching economy in telephony has 
been the elimination of operator assistance. When relay computers were installed in 
large numbers, the logical operations necessary for alternate routing, long-distance 
switching were feasible; Bell was able to introduce direct distance dialing. Elec- 
tronic switching systems (ESS) will allow switching centers to do jobs previously 
performed in other parts of the telecommunications system, such as: 

1. Assist in automatic dialing by maintaining a file of frequently called 
numbers for a subscriber. 

2. Route a call automatically to another terminal if the primary number 
is busy. 

3. Set up conference calls. 

The ESS has brought with it a steady reduction in the time a switch requires to make 
a connection. 
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A digital network, using time-division switches, may reduce modulation, 
switching and multiplexing costs for data communications. Three principal sources 
of cost savings for data users employing a digital communications system are: 

1. Elimination of conversion equipment presently required on their terminals 
to adapt digital signals for transmission over an analog network. 

2. Both local and toll switching would be l/3 to l/2 as costly if time-division 
technology were used. 

3. Assuming reasonable ranges of demand, time-division multiplexing is 
expected to be between l/lO and l/3 as expensive as frequency-division 
multiplexing, required in an analog network. 

The principal obstacle to a digital, multi-purpose network, serving inherently 
analog (voice, TV, etc. ) as well as inherently digital (data) customers, is the cost of 
converting analog signals into digital form for transmission over the network. 
Nevertheless, it may be possible to provide data users with a digital system, without 
either requiring analog users to adapt to a digital network or foregoing entirely the 
economics of joint usage between analog and digital communicators. The distribution 
lines of data customers could by-pass the local exchanges, which must receive 
signals in analog form, and enter separate digital, time-division switch. Signals on 
this digital subnetwork could enter long-haul transmission facilities directly, whether 
these were dedicated channels on transmission facilities shared with the analog 
network, or a special digital long-haul network such as might be provided by a 
multiple-access satellite system. 

Computerized store-and-forward switching is more costly than circuit 
switching. It is usually economic for activities requiring store-and-forward serv- 
ices to establish hybrid switching facilities, where a message is stored, if it requires 
processing or if the receiving terminal is occupied. Otherwise, a direct circuit- 
switched connection is established between the two terminals, and the message is 
relayed immediately. In most existing systems, a majority of messages can be 
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circuit-switched. The fewer the messages that must be stored, the less costly the 
the system will be. 

Just as the time-sharing systems are most costly when they must accommodate 
a broad community of users, so do the costs of a message-switching installation 
grow in response to diversity, i. e. , different data-speed, response-timing, and 
storage requirements. Moreover, when customers with incompatible terminal equip- 
ment share a single switching system and seek to communicate with one another, a 
larger share of their messages must be stored and forwarded, thereby generating 
cost increases. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

Reese, Frank D. , "Switching Trends Lead Independents into Digital Systems, Elec- 
tronic COs, " Telephone Engineer and Management . Vol. 78, No. 2, January 15, 1974, 
pp. 82-85. 

The growth of the toll networks and the acquisition of toll plants within the 
Independent Telephone Company's operating area, plus a consolidation of several 
smaller toll centers into larger ones, has in recent years had the effect of greatly 
increasing the toll switching machine size requirements. The anticipated trends are 
that those larger machines whose switching requirements in 1975 will be between 
5000 and 100, 000 CCS, serving from about 300 to 6000 toll trunks, will by 1990 be 
expected to handle between 18, 000 to nearly 1, 000, 000 CCS, serving between 1000 
and 55, 000 trunks. 

The next changing trend observed in the past few years is in the very rapid 
growth in the use of PCM type carrier equipment, often referred to as the "T" type. 
The number of channels is increasing at the rate of something on the order of 
400, 000 to 500, 000 channels or greater, per year. 

Probably the most comprehensive and recent view of the technological evolution 
taking place, and its impact on the telecommunications field, has been gained from 
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observations made at The International Switching Symposium held in Cambridge, 
Massachusetts on June 6-9, 1972. The following observations were noted. 


1. Stored Program Control . There existed an almost universal swing to 
stored program control. It was observed that the trend had shifted from 
a mild acceptance of stored program control some 3-5 years ago, to an 
almost complete acceptance not only by the development laboratories, 
but very importantly, by the telephone administrations. It was also 
observed that programming for switching machines is vastly different 
from programming for computers. 

2. Telco Acceptance . It was observed that national telecommunication 
organizations were now talking about the more rapid introduction of 
electronic switching systems into their operations. This acceleration 
appeared to be more rapid than indicated in the past. 

3. Device Acceptance . There was also less speculation given on the 
device operation (i. e. , did it really work? because the "state of the art" 
had reached the point where the devices could be counted on to perform 
their allocated tasks in a reliable fashion. However, one notable excep- 
tion to this was in the field of memories where much interest and 
notoriety was given to the use of semiconductor memories, and in some 
cases, improved ferrite core memories (improved in operating charac- 
teristics and, very importantly, in the costs of the device). 

4. Programming Language . A move toward utilization of higher level 
programming languages was noted, and this is prompted by the fact 
that programming manpower can be saved, as compared to the use of 
assembly language programs. However, it was also seen that with 
the use of higher level "MACRO" languages, more real-time of the 
machine would be utilized and more memory space would be required — 
and this increased the cost of the product. The caution, then, was to 
consider a judicious use of higher level programs, but at the same 
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time strive for greater programming efficiencies in those cost- 
bearing areas. 

5. PCM Switching . Another trend observed at the symposium was the 
almost universal move toward PCM switching, especially in the 
"transit switch" area where the savings are so noticeable. With the 
move toward integrated PCM switching/transmission in the tandem 
switching portion of the switching hierarchy, it was apparent that the 
technological fallout would next be applied to the Class 5 environment. 
However, it was also thought that the local plant would probably be 
space-divided, at least that portion facing the telephone sets, for the 
next several years. 

6. Compatibility . Switching systems planned for the future must com- 
municate with existing plant; the new systems cannot exist in a 
vacuum. This is because of the tremendous amount of investment 
in plant which cannot be ignored and must be integrated within any 
new switching plans. 

7. Maintenance and Administration. A final trend noted was the cen- 
tralization and maintenance of administrative functions. Technical 
Assistance Centers (TAC), as implemented by AT&T or their 
equivalent, were prevalent simply because this was the only eco- 
nomical way to cope with the very difficult maintenance problem 
facing not only the Bell System administrators, but all electronic 
equipment users in future. 

With the increasing usage of T-type digital carrier facilities, plus the fact that digital 
switching concepts are now economically possible and very definitely practical, we 
see a "digital thread" woven through the fabric of switching development. This will 
permit the creation of a "family of systems" having intimate compatibility, and 
permit common maintenance and training methods to be employed by the telcos in a 
manner never before thought possible. 
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C.5 LOCAL LOOPS (DISTRIBUTION) 

Local loops perform the function of carrying signals between user terminals 
and the local exchanges. Despite the predominance of transmission costs in local 
loop expenses, technical progress has been slower in this area than in the tele- 
communications network as a whole. Moreover, many cities require the telephone 
company to lay its local distribution cables underground — a procedure for which 
current technology offers few prospects of significant cost reductions. 

Broadband cable systems are substantially limited today to television distribu- 
tion, but they may be an economic, attractive media for some point-to-point services 
as well. As a distribution loop, broadband cable represents a substitution of trans- 
mission for switching, carrying every signal to every terminal on the loop rather 
than switching signals only to the terminal for which they are intended. The cable 
system uses frequency-division switching, where a customer simply tunes in the 
frequency channel he wants to receive. 

When installing cable for television distribution, it is relatively inexpensive 
to provide additional transmission capacity for other services through increased 
cable bandwidth, including point-to-point voice traffic. Present TV cables lack a 
signaling capability which must be provided for point-to-point service; in the near 
term, such service would be too expensive for individual subscribers, but perhaps, 
attractive to a few communicators who distribute a large amount of information 
among a wide local audience. 

A broadband loop, equipped to carry digital rather than analog signals, may be 
substituted for the present local loop configuration. This configuration would have 
no centralized switching or multiplexing. All signals would be carried on a broad- 
band loop which. would have a dropline at each user's terminal. There, a time- 
synchronized gate would open during the customer's assigned time slot and pass a 
signal between the terminal and the loop. This system would have the full signaling 
capabilities of the present telephone network. Some costs may be saved over the 
present local loop facilities; considerable speculation is involved because this kind 
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of system depends so heavily on future trends in the cost of LSI circuits. At present, 
it appears that digital loop distribution systems will be installed subsequent to 1980. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

Powers, Robert S. (ed. ), The Digital Loop: One Approach to the Wired City . Appendix G, 
Staff Paper 1 - Part 2, President’s Task Force on Communications Policy, PB 184 413, 
NTIS. 

This appendix includes a list of some of the services which might be offered on 
a general purpose telecommunications system, particularly nonradiating systems in 
urban areas. It is recognized that a transition will be required from the present 
paired-wire system or present CATV systems to the general system, which is 
characterized by coaxial cables carrying multiple digital signals and time division 
multiplex (TDM). 

Not all, but a substantial number of, homes and offices in the nation could be 
reached by a digital multipurpose system by 1980. The discussion of this subject 
is based on a number of explicit assumptions as follows: 

1. A list of assumed needs for telecommunications service, which are 
categorized according to the information rate required and whether the 
service must be one-way or two-way. 

2. There is no major technology which will be used on a large scale in 
public telecommunications systems by 1980 which is not already under 
development or at least being explored at this date. 

3. Solid-state technology will be able to provide large scale integrated circuits 
(LSIs) of high quality at low cost and in large numbers. This is probably 
the key assumption in this list and the one least subject to control or 
precise forecasting. 

4. Government regulation and legislation will encourage, or at least permit, 
the incorporation of new technology into the public telecommunications 
system. 
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5c Both the U.S. population and the per capita demand for telecommunications 
services will expand significantly in the next few decades. (It is predicted 
that the building plant in the U. S. will double by the year 2000. The doubling 
time for the telephone network has been about 10 years for several decades. ) 

The nation’s telecommunications system has a vital role to play in the solution 
to many of our present major problems including education, transportation, air 
pollution, race relations, probing of public opinion, etc. For example, telecommuni- 
cations facilities which allow a better approximation of face-to-face contact between 
individuals and groups of individuals would greatly reduce the need for physically 
transporting these individuals. If the telecommunication system is to have an opti- 
mum impact on these problems, it will have to be very much more versatile than 
our present telephone-telegraph- radio-TV systems. 

A list of needs which might be met by future telecommunications systems 
follows in Table C-l. These items are categorized by certain required characteris- 
tics of the telecommunications systems which will meet the need. One characteristic 
of highest interest is information rate (bandwidth or bit rate), which assuming no 
compression (or redundancy removal), corresponds to the categories /bit rates: 

(1) low (voice), 50 kbps; (m) medium (videophone), 7 Mbps; (h) high (television), 

50 Mbps. A second characteristic of highest interest is whether the communication 
is inherently one-way (entertainment broadcasting), or two-way (telephone conver- 
sation). 

C. 6 TRADE-OFFS AMONG TRANSMISSION SWITCHING AND LOCAL LOOPS 

Because transmission costs are falling faster than switching costs, some inter- 
mediate switching will be avoided by creating more direct routes. Why route com- 
munications from point A to point C through an intermediate switching center (point B), 
if it becomes relatively more expensive to operate the switching center than to pro- 
vide transmission routes directly from A to C ? When A-to-C volume becomes large 
enough, it justifies a direct high-density transmission line between the two points, 
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Table C-l. Needs for Telecommunications Systems 


Needs (service) 


Characteristics 
Rate Way 


Telephone 

TV (commercial or private) 

Stored TV 
Radio Broadcast 
Videophone 

Record communications 

Facsimile 

Shopping 

Advertising 

Teaching devices 

Voting by the public 

Meter reading (utilities) 

Alarms (fire, burgular, 
system failure, etc. ) 

Emergency communications 

Banking 

Access to time-share computers 
Communications between computers 
Mobile communications 
Vehicle traffic control 


1 2 

h 1 

1 or m 1 

1 1 

m 2 

1 2 


m 1 or 2 

1 or m 1 or 2 

1, m or h 1 

1 2 

l 2 

1 1 

1 1 


1 

1 

1 

1, m or h 
1 
1 

m or h 


1 or 2 
2 
2 
2 
2 
2 
1 
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exploiting economies of scale. Such economies in transmission and switching will lead 
to the construction of larger switching centers. This has been shown by complicated 
calculations involving the relative costs of high- and low-density transmission lines and 
large and small switching centers. There is a pronounced trend toward consolidating 
switching centers. Bell is considering dropping one of the upper levels in the switching 
hierarchy, so that trunk lines which previously came together only after passing through 
intermediate switching centers will now come together directly. 

The advent of domestic communication satellites will significantly alter the 
trade-off between transmission and switching for some services. The best prospects 
for these satellites are where they can replace both the transmission and switching 
functions of the terrestrial network, rather than as a direct substitute only for ter- 
restrial transmission facilities. Satellites are likely to find greater application in 
the provision of specialized long-haul networks for users with unique requirements, 
particularly where either the information rate or traffic loading on a given route is 
subject to wide variations over time. 

C. 7 TERMINALS 

Terminals convert outgoing signals for transmission over the network; for 
incoming signals the process is reversed. Technical progress has given the telephone 
handset new capabilities and (through the introduction of the touch-tone phone) the 
ability to generate a data signal; however, the cost of the terminal function has 
remained essentially constant. Today, station sets account for more than a fifth 
of total network costs; as other elements of the network become cheaper while 
demands on terminals for new services come about, this fraction of costs is likely 
to rise. Labor expenses are a major portion of terminal costs. The charges for 
connects and disconnects under current procedures may be reduced by lessening the 
involvement of service personnel when customers move. Terminals may be simplified 
by having other parts of the system perform traditional terminal functions; e. g. , the 
ESS may store a list of frequently called numbers and thereby remove from the 
terminal the burden of providing automatic dialing. The terminal in the home is foreseen 
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as interactive, providing many new communications services for the family. It is 
anticipated that cable TV networks will make such a service practical. A breakthrough 
in terminal design and cost is required. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

Baran, Paul, "Potential Market Demand for Two-Way Information Services to the 
Home, 1970-199(y' Report R-26, Menlo Park, Ca. , Institute for the Future, 

December 1971. 

A new industry, information services to the home, is expected by some to reach 
revenues of $20 billion per year by 1990, yet, it is an industry that does not in fact 
exist and is not even widely recognized. The Delphi inquiry by which market forecasts 
for possible future have information services were obtained took place in 1970. In 
the first round questionnaire, respondents were given a list of 30 potential new home 
information services, as shown in Table C-2. The 30 items were designed to be 
suggestive of the major types of services currently foreseeable. For each of the 
items, panelists were asked to give estimates for the following eight service 
characteristics: 

1. Average dollar value of one conversation (low, median, and high estimates) 

2. Duration of one entire transaction from completion of dialing to disconnect 

3. Percentage of actual transmission of data or message 

4. Average number of transactions per month, per home 

5. Percentage of service home subscribers could be expected to pay 

6. Most likely year of mass introduction in the United States (earliest, median, 
and latest estimates) 

7. Percentage penetration of households 5 years later 

8. The most likely entrepreneur to offer this service. 
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In the second round, respondents were given the results of the first round 
questionnaire and were asked to reestimate all numerical forecasts in light of the 
other panelists' responses. Table C-3 contains the medians of the panel's responses 
for each service, as well as additional computed results. 

The results are subject to a number of qualifications. The definitions of the 
30 items as provided to the panel were brief and in some cases ambiguous; different 
respondents may have interpreted a given question inconsistently. Therefore, the 
forecasts should be taken as broad-brush indications rather than precise projections 
of the new industry's general character. There is overlap between some of the 
services; the expansion of one service may diminish another. Aggregate market 
estimates for each category of services probably contain a certain amount of "double 
accounting." However, the list of 30 is deficient through inadequate foresight, 
omitting such major services as home security systems and utility meter reading. 

The principal direction of the report was to possible uses of broadband cable systems; 
however, the services presented were originally considered in a context where there 
were no constraints as to the communications medium used. Some parts of the 
questionnaires did consider new communications networks and techniques for the 
1985 time period that could be used for the 30 services and other purposes. Thus, 
the technological constraints on the hypothesized systems, although not explicit, 
were to some extent considered by the respondents. 

Finally, it should be understood that the data represents only the potential 
market and not the actual market that costs or technology will permit. The estimates 
assume that the services can in fact be provided at costs people are willing to pay: 
these forecasts represent the demand function and not the supply function. More 
precise estimation of future sales for each service must await a clearer picture of 
how these services might be provided. 

Jones, Martin V. , "How Cable Television May Change Our Lives. " The Futurist, 

Vol. VII, No. 5, October 1973, pp. 196-201. 

Mr. Jones states that cable television (CATV) makes possible a vast range of 
two-way information services, such as computerized teaching, thousands of plays 
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Table C-3. Summary of Median Forecasts 


Service 

Average $ Value 
of One Conversation 
Low ‘Middle High 

Duration 
of Single 
Trans- 
action 
(min. ) 

Data 

Trans- 

mission 

Connect 

Time 

(%) 

Avg . No . 
of Trans- 
actions/ 
Mo . /Home 

1. Cashless society transactions 

$0.10 

$0.16 

$ 0.40 

0.75 

20% 


40 

2. Dedicated newspaper 

0.10 

0.20 

0.50 

10.00 

95 


30 

3. Computer-aided school instruction 

0.50 

1.50 

3.50 

30.00 

20 


20 

4. Shopping transactions 
(store catalogs) 

0.20 

0.50. 

1.00 

6.00 

40 


10 

5. Person-to-person (paid work at home) 

0.50 

1.50 

5.00 

20.00 

50 


60 

6. Plays and movies from video library 

0.60 

2.00 

5.00 

90.00 

100 


10 

7. Computor tutor 

1.00 

2.00 

5.00 

30.00 

20 


10 

8. Message recording 

0.20 

0.35 

1.00 

3.00 

75 


7 

9. Secretarial assistance 

0.35 

1.00 

3.00 

10.00 

60 


10 

10. Household mail and messages 

0.10 

0.20 

0.50 

2.00 

90 


25 

11. Mass mail and direct advertising 
mail 

0.10 

0.17 

0.50 

3.00 

90 


25 

12. Answering services 

0.10 

0.20 

0.50 

2.00 

80 


20 

13. Grocery price list, information, and 
ordering 

0.20 

0.35 

0.50 

5.00 

80 


15 

14. Access to company files 

0.30 

0.60 

2.00 

5.00 

65 


10 

15. Fares and ticket reservation 

0.20 

0.35 

0.75 

5.00 

50 


5 

16. Past and forthcoming events 

0.10 

0.20 

0.50 

4.00 

80 


10 

17 . Correspondence school 

1.00 

2.00 

5.00 

40.00 

85 


10 

18. Daily calendar and reminder of 
appointments 

0.10 

0.20 

0.50 

1.00 

80 


25 

19. Computer-assisted meetings 

1.00 

2.00 

5.00 

.30.00 

40 


5 

20. Newspaper, electronic, general 

0.20 

0.50 

0.75 

10.00 

95 


30 

21. Adult evening courses on television 

0 .60 

1.00 

5.00 

50.00 

95 


10 

22. Banking services 

0.10 

0.25 

0.50 

2.00 

60 


20 

23 . Legal information 

1.00 

5.00 

15.00 

10.00 

75 


5 

24. Special. sales information 

0 .20 

0.50 

1.00 

4.00 

70 


10 

25. Consumers' advisory service 

0.25 

0.50 

1.00 

5.00 

70 


10 

26. Weather bureau 

0.10 

0.20 

0.50 

1.00. 

90 


20 

27. Bus, train, and air scheduling 

0.10 

0.20 

0.50 

1.25 

80 


5 

28. Restaurants 

0.10 

0.20 

0.50 

3.00 

SO 


5 

29. Library access 

0.50 

1.00 

2.00 

10.00 

90 


5 

30. Index, all services 

0.10 

0.20 

0.50 

3.00 

80 


10 
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Table C-3. Summary of Median Forecasts (Cont’d) 


% of 
Service 
Home Sub- 
scriber 
Expected 
to Pay 

Most 
of : 

Likely Year 
Introduction 

% Penetra- 
tion of 
All U.S. 
Households 

Median 
Transmit 
Time 
(min. ) 

Average Value 
of Service, 
S/Subscribing 
HouseholdAio. 

Value of Service 
After 5 Years, 
S/Average U.S. 
Household/Mo. 
(At Penetration 
Hates Shown) 

Early 

Middle 

Late 

Median 

Hich 

25% 

1975 

1980 

1990 

20% 

0.19 

$ 6.4 

$ 12.50 

$ 1.00 

75 

1980 

1983 

1990 

10 

9.00 

6.0 

15.00 

0.54 

50 

1975 

1982 

1987 

10 

10.00 

40.0 

100.00 

3.75 

25 

1977 

1985 

1990 

10 

2.50 

3.0 

5.00 

0.38 

5 

1980 

1985 

1990 

5 

6.50 

75.0 

250.00 

3.20 

80 

1975 

1980 

1985 

10 

90.00 

20.0 

50.00 

2.00 

80 

1975 

1980 

1990 

5 

6.00 

20.0 

50.00 

1.50 

90 

1975 

1980 

1985 

10 

2.50 

2.5 

6.25 

0.20 

100 

1975 

1980 

1985 

5 

6.00 

10.0 

25.00 

0.25 

75 

1980 

1985 

1990 

10 

1.80 

5.0 

12.00 

0.60 

0 

1980 

1990 

1995 

10 

2.55 

4.0 

15.00 

0.50 

100 

1975 

1980 

1985 

10 

1.60 

5.0 

11.25 

0.30 

50 

1975 

1980 

1990 

10 

4.25 

5.0 

7.50 

0.26 

1 

1980 

1985 

1990 

2 

3.50 

15.0 

50.00 

0.30 

40 

1975 

1980 

1985 

5 

2.50 

1.0 

2.50 

0.05 

50 

1975 

1982 

1990 

5 • 

2.70 

2.0 

5.00 

0.11 

75 

1978 

1984 

1990 

5 

30.00 

20.0 

50.00 

0.75 

100 

1980 

1983 

1985 

5 

0.95 

4.0 

10.00 

0.20 

40 

1975 

1980 

1985 

5 

6.00 

15.0 

91.00 

0.75 

75 

1980 

1985 

1990 

5 

9.00 

15.0 

22.50 

0.75 

80 

1975 

1980 

1985 

10 

45.00 

10.0 

25.00 

0.88 

60 

1975 

1980 

1985 

10 

0.85 

4.0 

10.00 

0.38 

100 

1980 

1985 

1990 

3 

7.50 

6.0 

25.00 

0.25 

80 

1975 

1982 

1990 

5 

2.70 

5.0 

10.00 

0.25 

100 

1975 

1980 

1985 

5 

3.50 

7.5 

10.00 

0.40 

100 

1975 

1980 

1980 

5 

0.90 

2.0 

5.00 

0.12 

80 

1975 

1977 

1980 

5 

1.00 

0.5 

1.00 

0.06 

60 

1975 

1980 

1985 

5 

2.50 

1.0 

2.50 

0.05 

100 

1980 

1985 

1990 

5 

9.00 

5.0 

10.00 

0.25 

50 

1975 

1980 

1985 

5 

2.50 

3.0 

5.00 

0.10 


$ 20.12 
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and movies available whenever one wishes, and electronically delivered newspapers 
edited specially for the individual reader. Many communications experts believe that 
CATV will drastically change human life in a wide variety of ways. Table C-4 sum- 
marizes the impact of interactive television on life as projected by Mr. Jones. 

Further consequences beyond the first-level effects are categorized in the 
following six headings. 

1. General 

• Greater sense of community awareness in small, geographically 
isolated communities. 

r < 

• Increased volume of Societal communications: within business and 
government, within communities, among people at all levels. 

• Reduced sense of national cohesiveness resulting from a fraction- 
alization of TV audiences. 

• Increased national vulnerability potential from action by dissident 

j ■ 

internal groups or from enemy sabotage. 

• More equitable availability of societal resources: educational 
opportunities cultural facilities; opportunities to run for public office; 
etc. 

• Decentralized siting for education. 

• Reduced dependence of suburbanites on the inner city. 

• Increased potential for invasion of privacy. 

2. Economic-Industrial-Occupational 

• Stimulated GNP, plant investment, business revenues, and employment. 

• Major industry realignments — over the air TV, movies, etc. 

• Reduced volume of business travel. 
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Table C-4. The Impact of Interactive Television on Life 


Applications 

Business-Work at Home 

Computer-Assisted Meetings 

Electronic Mail 

Business-Commerce 

"Cashless Society" Transactions 
"Dedicated" Newspaper 

Political 

Nationwide Voting Surveys 
and Voting 

Free Political Channels for 
Candidates 

Government 

Index of government services 
Vocational Counseling 

Health 

Remote Diagnosis 

Emergency Medical Information 
Household 

Consumer Advisory Services 
Water, Electric, and Gas 
Meter Reading 

Education 

Adult Evening Courses 
Library Access 


Art- Level Effects 

Quicker, more efficient management 
communications 

Better interface between different 
business firms 

More convenient shopping 

Better informed business and 
professional personnel 

More responsive public officials 

More candidates in local elections 


More equitable dissemination of 
authorized public services 
Less unemployment due to skill 
deficiencies 

More widespread distribution of scarce, 
expert medical knowledge 
Fewer accidental deaths 

Less consumer victimization 
Slightly reduced utility rates 


Better trained workforce 
Better informed citizenry 
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• Greater availability of electromagnetic spectrum for non-TV uses such 
as "land-mobile users" (e.g. , police and fire departments, taxis, etc.)* 

• Increased demand on the local level for TV programming personnel, 
cameramen, control board technicians, etc. 

• Shift in local advertising from newspaper to TV. 

• Reduced role of postal service in servicing the nation's transactions 
and correspondence needs. 

• Greater pressure on over-the-air TV to rectify its shortcomings — 
e.g., limited consumer choice, poor quality reception, etc. 

Political 

• Heightened political "infighting" at all levels concerning who is to 
regulate cable TV — federal, state, or local governments and executives 
vs. legislative bodies. 

• Increased lobbying for local cable TV franchises. 

• Easier, more equitable access to voters by all political candidates 
regardless of financial backing. 

• Greater trend toward "participatory" government at the expense of 
"representative" government. 

«■ 

Government: Administration 

• Need for new quasi-public institutions: 

a. To promote the use of public-access channels 

b. To monitor access to the channels 

c. To represent public in rate formulation 

d. To furnish "seed" money. 

• Reduced costs of providing basic community services — fire, police, 
postal services, etc. : 
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• Reduced problems in administering the "fairness," "right-of-reply, " 
and "equal time" doctrines. 

• Increased problems in controlling defamation, fraud, and obscenity. 

• A large retraining need for government personnel on how to use the 
cable effectively in their work. 

• New financial problems, e. g. , how the poor of the nation will pay for 
their communications if cable TV replaces or degrades over-the-air 
TV. 

5. Legal 

• Pressure for a review of many major legal concepts such as "common 
carrier" concept, "unfair competition, " etc. 

• Complicated anti-monopoly problems such as cross-media ownerships. 

• Increased copyright litigation and mounting pressure for a major over- 
haul of the nation's copyright laws. 

6. Recreation-Cultural 

• More diverse TV programming and enhanced consumer choice. 

• Greater trend toward sedentary recreational activity. 

• Enhanced opportunities to upgrade skill levels for those who prefer 
active sports. 

• Greater patronage and community support for such activities as operas, 
concert music, plays, etc. 

• Decreased use of books, magazines, and other printed matters. 

• Possibilities of electronic equivalent of junk mail. 

C. 8 MOBILE RADIO 

The growth of mobile land radio has been limited by two factors: the shortage 
of available spectrum, particularly in urban areas where use is heaviest, and the cost 
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and size of portable receiver/transmitters. A trade-off may be made between spec- 
trum use and cost. The less stringent the constraints on bandwidth, frequency 
stability, antenna directivity, modulation, and other measure of spectrum use, the 
less costly the radio equipment can be. A system of mobile radios with multichannel 
capabilities (more costly) uses less spectrum in achieving a given service level than 
a system of single channel sets. 

The direction of technological advancement is to provide spectrum-saving 
features at a cost comparable to available sets. By 1980, receiver-transmitters 
that operate with narrower bandwidths and tighter frequency control are likely to be 
on the market, and 10-channel sets should cost little more than single-channel sets. 

A principal constraint on portable, multichannel mobile radio has been battery 
weight. The development of solid-state components and integrated circuitry should 
have made low-power sets practical. 

BRIEFS OF RECENTLY PUBLISHED RELATED SUBJECT MATTER 

Communications: 'Waves of the Future," Newsweek, May 20, 1974, pp. 88-90. 

Of the 115 MHz made available, an FCC ruling this month allotted 40 MHz to 
AT&T and the independent phone companies and 30 MHz to other companies for use 
in the land-mobile radio industry; the remaining block of the new spectrum is held 
in reserve to accommodate any new developments or inventions. While no one was 
completely happy, the ruling almost doubled the spectrum available to the companies. 
In 5 years, the $500 million market for mobile-radio equipment is expected to top 
the $1 billion mark. FCC Chairman Richard Wiley called the decision "one of the 
most significant ever made by the FCC. It offers the communications industry an 
unparalleled opportunity to virtually revolutionize the land-mobile field." 

Bell plans to hook vehicles into a nationwide system allowing them to contact 
dispatch officers anywhere in the U. S. And, using a "cellular" concept, the giant 
company will vastly increase the number of subscribers it can serve in a city from 
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300 to 200,000. To do this, AT&T will use the same frequency in adjoining areas 
but at very low power so that broadcasts will not interfere with one another. 

The other companies may expand dispatch systems for police, firemen, and 
other users. It will also persuade equipment manufacturers to turn prototypes into 
commercial products. 

"It's a horse race now," says Walter Sutters of the Federal Office of Tele- 
communications Policy. Motorola hopes to market its 3 -pound portable telephone 
that fits in a briefcase. GE's plans are indefinite, except to supply mobile-telephone 
equipment and convert its two-way radio to the new frequencies. With room to experi- 
ment, according to some experts, it won't be long before investors come up with 
a practical version of Dick Tracy's two-way wrist radio. 

C. 9 TELEVISION DISTRIBUTION 

The least costly means of distributing TV signals is the existing method: 
high-power transmitters and tall antenna towers. Little change is foreseen in local 
broadcasting technology by 1980; it does not appear possible to provide large numbers 
of TV channels (12 or more) to most viewers through normal over-the-air broad- 
casting means with existing industry structure and lack of available spectrum. 

In considering alternative distribution methods that might increase the number 
of TV channels, the most interesting technological developments are occurring 
coaxial cable, in millimeter-wave radio, and in satellites. 

Cable television is attractive in densely populated areas where cables need not 
be laid underground and where there are few natural barriers. Current CATV cables 
have the capacity to transmit 12 to 25 simultaneous TV signals, depending on distance 
and repeater spacing. Cable is excessively expensive for reaching isolated viewers, 
to install in major cities where streets must be dug up, or to cross natural barriers 
such as rivers and mountains. Millimeter-wave radio or satellites may be used in 
conjunction with cable to reduce the overall system cost of providing multichannel 
capability to each viewer. 
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Millimeter-wave (in the 18 GHz frequency range) radio techniques may provide 
economical multichannel primary distribution. Using such a system the entire VHF 
television spectrum (12 channels) can be simultaneously transmitted over distances 
up to 6 miles, using a 100 MHz bandwidth. Its most attractive prospect is serving 
as an alternative to coaxial cable: 

1. In an urban environment with one transmitter using a fan beam transmitting 
to many receivers 

2. For reaching outlying suburban areas 

3. In by-passing natural barriers. 

At present the higher cost of such systems limit their potential use to inter- 
connecting localized (block or neighborhood) cable distribution nets to a central 
distribution point. It is problematical whether or not receiver costs could be reduced 
sufficiently to make direct broadcast to individual users feasible. 

The direct broadcast satellite is cost-competitive only for reaching a very high 
percentage of television households from a single organization point; it is technically 
feasible now. The per household costs of the space segment are substantial until the 
number of houses sharing the cost is large. In addition, if all TV program distribu- 
tion were by satellite, the viewer would be required to make an additional investment 
in a receiving antenna, which is estimated to range in cost from $50 to $300 per 
location. 
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GLOSSARY 


Batch mode A technique by which items to be processed must 

be coded and collected into groups prior to 
processing. 

Central processing unit (CPU) The central processor of the computer system. 

It contains the main storage, arithmetic unit, 
and special register groups. 

Cross-impact matrix A technique for examining potential relationships 

between forecasted events. If there is a cross- 
. impact, the probability of individual items will 
vary with the occurrence or nonoccurrence 
of other items. 

Cross-support analysis A matrix display to determine the support effect 

of each item of a field on all other items. It is 
used to clarify complex relationships. 

Cybernetic ■, Descriptive of a control system that links human 

brain and nervous system with mechanical- 
electrical communication systems, such as 
computing machines. 

Decision tree A structured study of outcomes represented by 

branches with assigned probabilities and costs. 

As the various possibilities are considered at 
branch nodes, deterministic or probabilistic 
judgments are made. Evaluation is made by 
finding "best" route down the tree, i.e., work- 
ing backward through the developed tree. 

Delphi conference A Delphi procedure where conferees are linked by 

computer which provides essentially real-time 
discussion feedback, yet preserves panelists 
anonymity. 

Exploratory An approach to forecasting that is sequential from 

data of the past. 

Gaussian distribution The most important frequency distribution in sta- 

tistics. A symmetrical bell-shaped curve also 
called the normal distribution. 
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GLOSSARY (Continued) 


Gross National Product (GNP) 


Interactive mode 


KSIM 


Monte Carlo trial 


Normative 


Ontological 


Outliers 


Packet-switched data 


The sum of all goods and services produced and 
paid for by a nation - including services supplied 
by Government. 

Usually a real-time, on-line computer operation 
where there is a high degree of man-machine 
interaction. 

A simulation language developed by Julius Kane 
for logical expression of cross-impact concepts 
and capable of stating in a realistic and graphic 
fashion the interaction of competing variables. 

An iteration of the same forecasting model, each 
of which changes the inputs of the model in ac- 
cordance with random values drawn from the 
probability distributions of the inputs. 

An approach to forecasting that is "needs 
oriented." Having stated goals or objectives, 
the forecast works backward to the present to 
see what capabilities now exist or could be ex- 
trapolated to meet future needs. 

Investigating the nature, essential properties and 
relations of being, interpreted as self-generating 
or "inner-directed." 

Those values in a set of individual values that 
appear to differ considerably from the others 
(statistical). 

Digital transmission of messages by packets con- 
sisting of a fixed number of bits. Packets are 
transmitted when ready and assembled at desti- 
nation. 


Policy Delphi A Delphi study or exercise devoted to examining 

policy issues. 

Quartile The three points of a frequency distribution that 

divide the distribution into four parts. 
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GLOSSARY (Continued) 


Signal/Noise (S/N) ratio The relative strength of a wanted signal to that of 

the noise interference present, the ratio being 
normally expressed in decibels. 

Teleological The fact or the character of being directed toward 

an end or shaped by a purpose. 
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